Yes! I mean, No! I mean whatever is most opposite of your point that will annoy you the most!
Actually, as to your specific question, I agree with you. Everyone has been screwed over by someone at some point. No human group can claim the moral high ground, because somewhere in their past...