It is probably in your head. We naturally think every little thing happen because of US. Chances are you just need to suggest a place for people to go or ask where everyone is going for lunch. There are only three people in my office so it doesn't happen here but I can see different lunch clicks forming in a big office.
If it is happening, you probably said the wrong thing to the right people to get black balled from the "Cool" lunch crew.