homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Gold Sponsor 2015!
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Why is there a 301 redirect code for robots.txt
seeing a 301 code a few times each day for robots.txt
Mike McKnight

10+ Year Member

Msg#: 4027492 posted 6:34 pm on Nov 18, 2009 (gmt 0)

Why would this happen?
every day I see a 301 redirect when a crawler asks for my robots.txt file
I'm not sure it hurts but the behavior seems strange, I do not have two robots.txt files, never had.

There are two lines from my server logs below, 1st line shows the the GET request with the 301 code followed a few lines later but a GET with 200 success code

This is not a consistent happening as most requests are honored with a 200 success code.

crawler5138.ask.com - - [17/Nov/2009:01:58:03 -0500] "GET /robots.txt HTTP/1.1" 301 567 "-" "Mozilla/5.0

crawler5138.ask.com - - [17/Nov/2009:01:58:09 -0500] "GET /robots.txt HTTP/1.1" 200 5227 "-" "Mozilla/5.0


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved