It doesn't really make sense, since they play their games in New Jersey, but its more of a marketing thing. I mean, the New York Giants sounds better than the New Jersey Giants. New York is simply a larger market. Its kinda the same thing that the Angels did when they changed their name to the Los Angeles Angels of Anaheim, despite not being in LA.
The other thing is that they were originally from NY, when they played in Yankee stadium. They even moved to CT for a few years when they played at the Yale Bowl. When they finally got their own stadium in Bergen County, its so close to NY City (you can take a bus to the city for like $3) they decided to just keep the name NY.