Your man is an idiot. You are who you are and at least you want sex, right? So, quiet is fine. Actually I do not like noisy, it is irritating to me and stupid so I think you are o.k.
I would though, with that said, talk to him about what you want while you are in a sexual situation as that is...