It seems only to be interested in the newest pages created in my case. Still would be nice to know what this is for. It has been "testing" for several months now.
quadcity
10:30 am on Aug 19, 2005 (gmt 0)
>It seems only to be interested in the newest pages created in my case. Thanks for the information. I'll study my logs to see If I notice anything else.
jahfingers
11:47 pm on Aug 24, 2005 (gmt 0)
Just saw this for the first time today -- returned a 404 when looking for a robots.txt file that is there?
volatilegx
1:24 pm on Aug 25, 2005 (gmt 0)
The 404 error would have been generated by your web server, not the spider. Are you sure the robots.txt file is in the place where the spider thought it was? Was the spider asking for robots.txt in the / directory?
