You're right.
I'm not very careful about what sites I visit, but I care about security, since like most people I use the computer for everything including banking etc. But I have very little faith in anti virus software , I think Jonh Sabugs is very right on the money. However I do use it because it does no harm either. (This means I don't use the kind that ask you about stuff, I tried them in the past but I don't think they offer better protection after all.)
Anti virus software is a minor and auxiliary part of computer security. It's like in a movie, that tiny object someone carries in his shirt pocket, and when he's shot it stops the bullet and saves his life. Well that means he got lucky, not that it was an adequate protection. (Although I'm not saying the odds are comparable, it's just an illustration.)
Remember those hated UAC popups in Windows Vista (which were relaxed in Windows 7 because of consumer feedback)?
http://www.winsupersite.com/article/...ccount-control
Disabling this Windows feature is the single most damaging move you can make against your own security. Installing anti virus software while having this disabled (like most people do) is like walking around in the jungle where any wild animal can eat you, but wearing a helmet. And some of the people who complain about UAC being too annoying use one of the very popular client firewalls, which are pretty much useless, and far more annoying.
I could go on, but the point is that people's idea that having anti virus software makes you safe and not having it makes you unsafe, is radically wrong.