Actually they can improve your image quality.
UV light scatters and is not focused in the plane on your film. If your film reacts to it the result is a slight foggyness in the image.
With the polarization you are cutting out glare and reflections. It will let you get the image of a car instead of the reflection off the car's finish, or let you photograph through glass if you need to.
It will not eliminate all glare and reflection but it will cut out a lot of it.
As with all parts of the lens system try and keep them clean. Dust interferes more with the image quality than most people realize so treat them with the same care as the lens itself.
They are not substitute lens caps!
They should be regarded as part of your lens system, and you should be using lens caps with them.
(but yes, they do offer a bit of extra protection to the lens itself)