What do I do about my family situation,them hassling me about my business?

[BuTtErCuP]

Ok I am fixing to turn 28 and am with a older guy that is 42. We've been together 4yrs now. He makes me very happy. Thing is my family recently found out who I'm dating and don't approve. I live with my kids at my mom's. I have always been secretive with my business from my family. I'm the black sheep and they always look down on me. They keep being negative saying awful things. They just know who he is but don't really know him,never even met him! They say they hear he's a woman beater and such and such. He never hits me but of course they believe what ever. When I get home late they think I'm on drugs and I don't do any of that neither does he. They say they want me to text them when I go out to let them know where I'm at,with who and what time I'm getting home. That is crazy because I'm no kid! They do not even pay my phone but for some reason every single time when they see me on my cell they have someting to say. They say I'm on it too much or on the computer when I pay for all my stuff,bills and kids things! I am a good mother and all my friends ate always telling me I'm not doing anything wrong they need to leave me alone. I have my own car and everything but they still think they can run my life. When I don't even go out but my man does he stops by on his way home. We don't do anything I've never bought him around my family. I just go outside and talk for a little bit then he leaves. I don't see nothing wrong with that. As grown as I am I have to sneak out just to go stand outside with him! When ever I'm in my room my mom trys to listen to my conversation! I am really depressed staying hear. Thing is that I lost my job a while back. I am surving off my kids support in the mean time. As soon as I get on my feet I want to move out and away. I need my own space so there for if I want to have my man over that will be my business. What should I do in the mean time? And do you think I am right or not?