I'm only about 5'6 so I'm usually the smaller shorter guy. I weighed about 160 before I started doing Judo. I started boxing in high school but was never really good. When I got to college, my suitemate who is about 5'8 /130lbs grabbed me by the collar, executed a beautiful Judo throw, and transitioned into an armbar. Since that day, I have been practicing Judo regularly. When I transferred schools, I got into MMA at a wrestling based MMA gym with focuses on Muay Thai and BJJ as well. I am one of the only amateurs at the gym as many of them fight on pretty big promotions. Some of the guys who are contracted with the WEC are 5'5-5'6 and I have no doubts about them taking down a big guy.
In my only fight in the real world, I used a Judo throw called Osoto Gari. The guy was about 6'2 and maybe 190 lbs. Since he was only intimidating and had never practiced and martial art, his balance was broken easier than some of the 12 year old girls in the child Judo classes. The throw was on grass so I figured it would stun him and I would transition into an armbar or throw some hammer fists to his face afterward. However, not knowing how to fall, he landed so hard on his back that he lay there squirming in pain. I just left.
Still, don't be fooled by these guys who say size has nothing to do with it. Size is definitely a factor. Especially when it comes to guys like the Kung Fu guy who talks about "one good strike" to put anyone down. By all means, kick a guy in the balls but don't think you're going to execute that glorious TKD, Karate, or Muay Thai strike and a bully is going to be done. It just never seems to work out that way.
My advice would be to learn a grappling form in addition to striking. On the ground, people who don't know what they are doing will lose 99.9% of the time. I'm only a BJJ blue belt and when new people show up, its amazing how easy it is to tap them out. But when I spar striking with those same people, they get punches on me every once in a while. Don't risk it in a street fight. That big fat guy might not train martial arts, but his hands might be battering rams. Thinking you are Bruce Lee and able to just drop dudes at the snap of a finger is silly.
I also have a friend who is 6'1 and weighs 275. He is straight up fat. Still, in the one fight hes ever been in, he ate a shot to the face, bear hugged the guy, picked him up, and threw him down a muddy hill. I'm sure these other guys will say that if he tried to do this to them, they would kick him in the gut or punch him so hard that it wouldn't happen. My answer to that is...sure, maybe. But who knows. Some people have iron chins, some people have fat strength, some people have retard strength. Conventional wisdom says not to fight a bear even if its mouth is muzzled closed. Why? Because its bigger and stronger than you.