While your doctor or other posters may say it is just fine, I say don't take the risk. Inhaling paint fumes is no different from huffing, IMHO, and you are taking an unnecessary risk. Heck, even not pregnant, I get dizzy and get nauseous from paint fumes. You KNOW that stuff is getting in your system.
I also disagree with posters who say that you should just use a mask. The masks that actually prevent inhalation of fumes are VERY expensive. You could hire someone to do the work for you for the cost of those masks.