They are probably not hurting the plants. If you have really hundreds of them, there might be a nest in your garden. I have got rid of ant nests with a 50:50 combination of boric acid (borax) + icing sugar. You can put it in a little yogurt container or the equivalent, with a lid & a few holes. If they're carpenter ants (big black ones) you should mix it into a liquidy paste with a little water.
Maybe more effective, is Ant-B-Gon liquid (try a hardware/home depot type store). Again, a few drops in a yogurt container with some holes so they can get in. You can add a wee bit of icing sugar/borax to that as well to help attract them.
In both cases, the ants are supposed to carry the stuff back to the nest where the other ants will clean it off them & be poisoned.