Simple equation to figure it out:
If your love of this guy is greater than your concern for what his family thinks, stick with it.
If your concern for what the family thinks is greater than your love for this guy, the end it.
At the end of the day, it's a big difference, but not so big it...