Hi, I've got a simple spider I've made using Perl. I've got it to check robots.txt using the code below...
$browser = LWP::RobotUA->new(
'Fred' , 'me@email');
$browser->timeout(10);
my $response = $browser->get($URL);
die "$URL -- ", $response->status_line, "\nAborted"
unless...