Douglas was the incumbent Democratic senator. He was well-known, and many thought he would ultimately become President. Lincoln was not as famous, and he belonged to the newly-formed Republican party. Each candidate exaggerated the extremism of his opponent. Lincoln was not the rabid abolitionist Douglas claimed, for example.
At the time in Illinois, U.S. Senators were elected by the state legislature, not by a popular vote. The Democrats had a majority in the Illinois State Legislature, and they re-elected Douglas to his Senate seat.