I would contact Legal Aid. It's probably just a threat. They have more to lose than you do. If the court sees no contact or interest, they will probably still give you custody plus make the father pay child support.
I would get legal counsel to protect yourself, but not allow them to...
We are renting a house with a garden that clearly used to be a well-tended, beautiful, formal garden. I want to take up gardening and learn how to care for our new garden. Now, it is over-grown and in disarray with huge clumps of the strong (bleeding hearts, lillies of the vally, etc.) choking...