Which is better for making a fast and efficient web spider? Python, C, Perl, or other?

N

nicholasdewaal

New member
I want to collect specific parts of several web pages that are changing rapidly--thus requiring heavy use of regular expressions and text analysis, and the need for speed.
 
I think the actual http request to get the information is going to take a lot more than your actual processing. And you don't want to make a lot of requests per second to a website because you could end up doing an unintentional DoS attack (which can be considered trespass to chattels and you'll have a lawsuit coming right up). There should be a crawling delay of about 10 seconds, in which you would be able to do a lot of processing. So better choose the language you're most comfortable with.
 
Perl is perfect for that. There are DOZENS of pre-made libraries for doing what you describe, so you will have to write VERY LITTLE code yourself!
 
Back
Top