hit a car couple of weeks ago?

[Sal Lecter]

I hit a car a couple of weeks ago and left the scene of the accident. When i came back and parked in the same place a few weeks after the accident, i received a letter on my car that said that i had hit his car and here was his number. Does he have any real case against me? There is no paint from my car on his, just a dent in his car. and my car has some scrapes but nothing that would lead anyone to believe caused his type of damage. He didn't file a police report and hasn't come face to face with me to talk about it even though Ive seen him countless times face to face. Now that he most likely has my license plate number since i parked in the same place as last time since the accident was a few weeks ago and he didn't file a police report and is only now trying to confront me on it. is this a my word against his word kind of thing? can he actually do anything about this?
also he technically didn't really see me hit the car. he heard it, and saw me down at the end of the block by the time he got to the street.
if i was caught then why hasn't he come told me face to face then? obviously he knows the house i am in yet he doesn't ring the door bell?