For God's sake. "I may have to start watching Conan now". If I didn't disagree with using that horrible 'rolly-eyes' smiley, I'd have used it there.
He hasn't killed anyone. He has had consensual affairs with women, of above legal age, who work on his show. There's no evidence that he coerced anyone, or that's he's a sex pest. In fact, Stephanie (alleged to be one of the women concerned) is said to be mortified about the whole thing, and is obviously still on good terms with Dave as the affair ended in 2003 (before his son, Harry, was born) and she still works on the show.
It's nobody's business apart from Dave's and Regina's. It doesn't affect his ability to do his job in my opinion. It's not like he's Roman Polanski who raped a 13 year old and apparently has half of Hollywood calling for his release on the basis that it was a long time ago and he's made some good films.
If Letterman does lose his job over this, then it's an absolute disgrace, and those people who say they're off to watch Conan or Jay Leno instead - fine, do it! Your loss, frankly.