Ask your gynecologist about it, If you don't have one just go to the doctor's and they answer any questions.
Doesn't sound like anything to get to worried about, unless it hurts. I don't get that when i use condoms. I just have a strong odor the first time i pee after having sex with a condom which is probably just the lube that is on it.
You could just try other forms of birth control, like spermicidal lube, daily pill, ring, the shot (a shot that stops you from having a period therefore not ovulating), a diaphragm, or the patch. You can also use two (Depending on which two, don't use the patch and the pill!) but you can use the diaphragm and the lube to lower your chances or getting pregnant.
Look into other methods, you might find one you like.