Forum Moderators: goodroi
There are two lines from my server logs below, 1st line shows the the GET request with the 301 code followed a few lines later but a GET with 200 success code
This is not a consistent happening as most requests are honored with a 200 success code.
crawler5138.ask.com - - [17/Nov/2009:01:58:03 -0500] "GET /robots.txt HTTP/1.1" 301 567 "-" "Mozilla/5.0
crawler5138.ask.com - - [17/Nov/2009:01:58:09 -0500] "GET /robots.txt HTTP/1.1" 200 5227 "-" "Mozilla/5.0