Hi, I've got a simple spider I've made using Perl. I've got it to check robots.txt using the code below...
$browser = LWP::RobotUA->new(
'Fred' , 'me@email');
$browser->timeout(10);
my $response = $browser->get($URL);
die "$URL -- ", $response->status_line, "\nAborted"
unless $response->is_success;
print "$URL let me in!
\n";
If this bit of code is NOT in the spider, it connects to the web site very quick, but if it IS in the spider, it takes ages (about 5 mins) to connect. What bit would I have to change/take out to speed up the connection?
Thanks for your help.
$browser = LWP::RobotUA->new(
'Fred' , 'me@email');
$browser->timeout(10);
my $response = $browser->get($URL);
die "$URL -- ", $response->status_line, "\nAborted"
unless $response->is_success;
print "$URL let me in!
\n";If this bit of code is NOT in the spider, it connects to the web site very quick, but if it IS in the spider, it takes ages (about 5 mins) to connect. What bit would I have to change/take out to speed up the connection?
Thanks for your help.