Road grime on the frame means nothing. But you figure a person who leaves a dirty frame isn't going to be to concerned with the bearings of the headset, hubs, and bottom bracket. And that will effect the performance of your bike.
If you look good you feel better, feel better ride better, ride better you can leave others behind you saying "they look good"... so all in all most people ride better with a clean bike. If you have sealed bearings you have nothing to worry about.
The surface area would be too small to notice any difference. A little road grime wont make much difference. Myth Busters did a test on a dirty cars versus a clean one and it wasn't conclusive. They then covered the car in clay and put dimples all over it like a golf ball and seen a pretty good gas mileage increase. I noticed in the Tour De France in the time trial one on the riders had the golf ball like dimple's on their helmet. You can try it dirty and then wipe it clean and find out.
The theory behind the dirty car being more fuel efficient was that it broke up the air flow more and reduced the drag.
Well i suppose if you look at it scientifically the clean bike would be faster than the dirty bike. The gears would be better lubricated and that would ensure a better movement. The dirt would increase drag and weight and i know you are not talking about mud etc but I am talking scientifically. The difference would be minimal but there would be a difference.