Yes! I mean, No! I mean whatever is most opposite of your point that will annoy you the most!
Actually, as to your specific question, I agree with you. Everyone has been screwed over by someone at some point. No human group can claim the moral high ground, because somewhere in their past, their ancestors did something to someone.
However, as deporting illegal immigrants, I think first you have to stop employers from offering them enticements to cross the border.