No. Not because it's an exclusive club or because you're doing anything wrong, but just because vegetarian is a word with a dictionary definition.
Just as teetotaler is a word with a dictionary definition; so someone who drank no alcohol except gin, and the only gin they drank was Gordons gin, couldn't claim to be a teetotaler.
And people who eat fish but call themselves vegetarians can make life difficult for vegetarians. Every vegetarian will probably have had a similar experience to this: at a catered function - conference, wedding, work-related course - a vegetarian option has been promised. It turns out to be fish. The caterers reply to vegetarians' protests by saying that other vegetarians they've catered for have been happy to eat fish.
