how does size matter in a fight?

[The Mind's Eye]

some people say size doesn't matter in a fight but obviously it does matter or else boxing and stuff wouldn't have weight divisions... so i was wondering what way does size give an advantage in a fight? how about disadvantages?
but wouldn't a smaller person have more speed/agility/endurance?