SD cards work differently from hard drives. With both a hard drive and an SD card, data gets fragmented when you delete small files and then save larger files to the disk. The large file fills in the small gaps left from where you deleted the small files from.
On a hard drive, this sucks, because the drive heads have to physically move around over the disk to find all the bits of the file. As it takes a short time to move the disk heads it means that reading and writing files becomes slower over time, until you either defragment the disk or format it and copy all your files back on.
An SD card is different, however. It has no moving parts, and it takes no more or less time to access a highly fragmented file as it does a non-fragmented file. So SD cards don't slow down over time, and formatting them or degfragmenting them won't speed them up at all.
However, you can speed things up by deleting unnecessary files. Android may well have to read the names of all the files in a given folder to access a file. So the more files you have in that folder the more time it may take. It just depends how it's accessing the files.
You're stuck with FAT32 to Android, so don't bother trying NTFS or anything else. Also FAT32 causes less wear and tear on an SD card as it's a non-journalling file system, so doesn't write to the SD card as much. SD cards have a limited number of writes before they die, so you don't want to be using them up too quickly. This is why flash cards and flash USB drives always come formatted as FAT32.
You can increase the speed of an SD card in other ways. Sometimes a dedicated card reader can be faster than USB through the phone. You can get a higher class of SD card. Class 6 is the fastest, class 1 is the slowest. Even with high classes of SD card it's worth noting that the class literally means the number of MB/s the SD card guarantees it can write. So a class 6 card only guarantees to be able to be written to at 6 MB/s.
Also make sure you're connecting to your phone via a USB2 port. Some ports, especially ports on the front of a desktop PC, are often USB1.
