I would just think either you were too lazy to get dressed, were in to that kind of style, or were coming back from the gym
people always judge people by how they dress, that's how it is
instead of complaining just start dressing in the way you want to be perceived