No, you shouldn't. The truth is, we've been genetically modifying food for millennia, however before it was from breeding. Now, we inject the exact genes we want into plants that bring about traits that we want. Otherwise, we would have to wait for the plant to develop the trait by random mutation, which could take a long time, or never happen at all. The actual problem is patented property, as companies may put a patent on their genetics. When a genetically modified crop produces pollen and fertilizes a non-modified crop, the resulting seeds will have that special gene, unknown to the farmer. At that point, he may end up using the seeds that contain genetic property of a company, and that company may find out and sue the farmer who never even knew what happened.
Also, to make weed-killing easier, we'll insert genes for herbicide resistance into our crop plants, so when we spray chemicals, they won't get hurt, but the unwanted plants (the weeds) will. Unfortunately, plants aren't as picky about genetics as animals, and so plants can pollinate to other species. This may end up giving the pesticide resistance genes to invasive plants, which means when you try to kill them with herbicides, they won't die, and become a bigger problem.
For the consumer, though, GM plants are perfectly safe, so don't worry.