It's often a taboo because sex outside of proscribed social constructs is a recipe for chaos and disaster. It complicates our understandings of relationships, sex (which we often conflate with love), the nature of monogamy and commitment, etc.. Not to mention that the potential children from a one-night stand pose a very real problem for most cultures, since we have invested concepts about parenting and the government is even involved many times.
As for the double standard for women, well, that goes back to archaic views of female sexuality and women's place- women used to not be autonomous, and were not believed to have sexual desires of their own. A woman choosing to have sex with someone she didn't know or care for intimately was impossible to conceive. Despite our best efforts, we are still haunted by this sort of thinking.