I think it's a real concern, but I think it's more likely to show up in the phrase "not averse to" than, say, "risk averse."
On TT, "not adverse to" gets 20 hits, while the correct "not averse to" gets 22 hits, in both cases spread over the branches. That's a pretty poor showing for TT'ers; an unscientific google search shows the correct usage outweighs the incorrect one by about 12:1.
A google search restricted by region (USA, Australia, UK) shows the wrong usage occurs about one-third of the time in the USA, one-quarter of the time in Australia and one-fifth of the time in the UK. So I am editing this post to remove the reference to wondering whether it's a big deal in the USA.
CK
Edited by: chriskean1