Yes by all means. If he did it once he will do it again. You didn't mention if drugs or alcohol were in play. Irregardless, this could become a trend.
Be up front with him and tell him your nervous about him becoming abusive with you. Move on, there are plenty of people out there who will never get physical with you. This is a bad start. Good Luck.
I AM IN THE PLAYPEN WITH SWEET MILDRED