Sounds like a genuine misunderstanding to me, if he thought you were a semi-gaylord as well.
Say to the bloke that you don't have a gay bone in your body and you certainly don't want his gay bone in your body. That you think he's an OK guy and that you would like to be blokey mates. That if...