| Welcome to WebmasterWorld Guest from 220.127.116.11 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
|Why is there a 301 redirect code for robots.txt|
seeing a 301 code a few times each day for robots.txt
Why would this happen?
every day I see a 301 redirect when a crawler asks for my robots.txt file
I'm not sure it hurts but the behavior seems strange, I do not have two robots.txt files, never had.
There are two lines from my server logs below, 1st line shows the the GET request with the 301 code followed a few lines later but a GET with 200 success code
This is not a consistent happening as most requests are honored with a 200 success code.
crawler5138.ask.com - - [17/Nov/2009:01:58:03 -0500] "GET /robots.txt HTTP/1.1" 301 567 "-" "Mozilla/5.0
crawler5138.ask.com - - [17/Nov/2009:01:58:09 -0500] "GET /robots.txt HTTP/1.1" 200 5227 "-" "Mozilla/5.0
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved