Is there a php script that takes a URL, and then tells the script whether the...

N

nelsongchen

New member
...URL should be read or not? This is with regard to complying with the robots exclusion standard (robots.txt) that a site may have. I'm trying to write a compliant bot, FWIW.
 
Back
Top