The idea of a cache is to store something that is normally retrieved slowly in a way that it can be retrieved much faster. I good example of this is the Market cache. It stores all the icons for any application you scroll past. The first time you see an icon for any application on the Market it will be loaded from Google's website. This speed is limited by your internet connection speed (i.e. 2G, 3G, WiFi, Broadband, etc). The Market app then "caches" the icon - which pretty much means it saves it to your phone. The next time you scroll past that application, the Market, it will load the icon directly from the phone, which is likely to be hundreds or thousands of times faster. Great right? At this level, it is.
Caches have some problems though. Firstly, they can grow to enormous sizes. There comes a point where you start to run low on storage, and this causes a number of problems in itself. So clearing the cache (deleting all those app icons you've ever seen in the Market) frees up a lot of space.
Secondly, the cache has an index which the Market will use to check to see if a given icon is in the cache. Most programs store their cache indexes in memory as this offers a huge speed increase. But, if the cache is big, the index is big, and hence it uses more of your precious RAM. With most modern computers (Windows boxes, Linux boxes, Android phones) as you start to run low on RAM things start to slow down, as the operating system starts having to use your disk (or in the case of Android your internal flash drive) as memory, which is much slower, and as such your computer/phone will slow down. Hence, clearing the cache if it's too big will free up RAM too, and hence speed up your device.
Also caches can get out of date. What happens if the developer of the app changes his icon? So the Market still has to do a small check over the internet to see if the copy of the icon it has in the cache is the current one. So, sometimes, you see out of date stuff when using caches.
Also, then the indexes get big, they take longer and longer to search through to find stuff. At some point, the searching of the index will take longer than it would take to just grab the icon off the internet.
The bottom line is, caches are great, but they can become too big, and it's always a balance. Big caches have a greater chance of the icon being already stored in the cache, but use lots of storage, RAM, and take a longer time to search. Smaller caches have a lower chance of the icon already being cached, but use less storage, RAM, and are quicker to search.
Obviously it's not just the Market that does this. Browsers are another great example, as are big commonly used database queries.
Some apps will self-limit the size of their caches to try and automatically address this. I have no idea if Android, or any of its apps, do this.
You're correct with the memory size versus task killers. It's very similar to the cache dilemma. Android effectively "caches" (and I use that term loosely here) apps that you've started, so that they start more quickly next time, at the cost of them taking up some of your RAM. Task Killers can kill them off, to free up your RAM, at the expense of the app starting more slowly next time. However, Android manages this itself. It will AUTOMATICALLY kill off processes when it runs very low on RAM. However, Android has the advantage that it will also attempt to restart killed processes when RAM becomes free again. Android also has an advantage in that it uses some common sense when killing off apps. It kills off the ones that aren't needed, aren't being used, don't need to run in the background, etc, first. Task Killers, at least most of them anyway, simply kill stuff off, even if it's a vital Android component that is being actively used, and they never restart them.
I used Advanced Task Manager. It has a blue gear icon. I don't know if it's the best, but it works very well for me.
And after saying all that, why do I use a Task Killer (I hear people asking)? Well there are some apps that I do only run once in a blue moon, so there's no advantage to having them remaining running in the background ready to start more quickly. There are also some apps that start themselves and I have no control over them - and they suck power. ShopSavvy does this to update its GPS position every hour or so. I don't want it doing that, so I kill it off.
The term "running" just means that the application, or widget, is doing its thing (killing tasks). I guess it depends if the task killer is an app or a widget. I use the app part of the task killer I use, and don't use the widget as I don't need to have it on my home screen all the time taking up space. I just run the app whenever I want to kill stuff off, as it's very rare I need to do it.
Most task killers run automatically every X minutes. Most have a list of apps that YOU need to make, that decides whether an app will be killed off (a kill list), or excluded from killing off (an exclude list). I don't have the one I use running automatically, but it can do that.
I don't really know what the widgets for task killers do. I've never used them. But I think the one I use is simply a button that allows quick access to the task killing process, rather than having to wait X minutes for it to do it automatically.
