Well, it depends on your reasons for going vegetarian. If you care about animal suffering, you're obviously aware of the fact that fish feel pain and die slowly (by suffocation, either with or without decompression and/or having a hook stuck through their face or gut). If it's the environment that gets you going, you should be aware that nets dragging along the bottom of the ocean can cause serious problems as well, destroying coral, disrupting bottom-dwellers, and stirring up mud. Commercial fishing has also done a number on animal populations and biodiversity. Just run a Google on "commercial fishing and the environment" and you'll find a ton of information about it.
You managed to give up other animals and foods that they make, even though you probably enjoyed cheese and maybe liked chicken noodle soup, bacon, burgers, and other stuff that you've given up. Giving up fish will be a challenge, but it's probably one that you'll want to face depending on your reasons for skipping other animal products.
