I'm from a country with actual histo...I mean, I'm not from the US, either.
Now, "weebs" - well, that's different. If you walk around with your waifu-pillow and talk non-stop about how 2D girls are better or somesuch I'd probably feel the need to punch you, too.
But that doesn't have anything to do with just liking anime/manga or watching some. If you're annoying about something you like to somebody who isn't that interested you'll always get negative reactions, that's just common sense. I wouldn't want to hear about which wheel arrangement is the greatest for steam locomotives either (although that's clearly 1E, unless you're English, then it's 2-10-0).