Please tell me you got something in writing saying it was a gift and not a loan...if not don't worry too much about it. Unless he has something in writing saying it was a loan and not a gift, he's in the same boat you are. In order for him to sue, you have to have had an agreement to pay it back in first place.
In the future, don't accept gifts of money from people, make your own way, or borrow from a bank. Free-loading nearly always has a negative outcome for someone involved, either the free-loader (in this case you) or the one paying for stuff (your ex) gets burned.