| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Gold Sponsor 2015! |
|Why is there a 301 redirect code for robots.txt|
seeing a 301 code a few times each day for robots.txt
Why would this happen?
every day I see a 301 redirect when a crawler asks for my robots.txt file
I'm not sure it hurts but the behavior seems strange, I do not have two robots.txt files, never had.
There are two lines from my server logs below, 1st line shows the the GET request with the 301 code followed a few lines later but a GET with 200 success code
This is not a consistent happening as most requests are honored with a 200 success code.
crawler5138.ask.com - - [17/Nov/2009:01:58:03 -0500] "GET /robots.txt HTTP/1.1" 301 567 "-" "Mozilla/5.0
crawler5138.ask.com - - [17/Nov/2009:01:58:09 -0500] "GET /robots.txt HTTP/1.1" 200 5227 "-" "Mozilla/5.0
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved