Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Why is there a 301 redirect code for robots.txt

seeing a 301 code a few times each day for robots.txt

6:34 pm on Nov 18, 2009 (gmt 0)

New User

10+ Year Member

joined:June 30, 2004
posts: 36
votes: 0

Why would this happen?
every day I see a 301 redirect when a crawler asks for my robots.txt file
I'm not sure it hurts but the behavior seems strange, I do not have two robots.txt files, never had.

There are two lines from my server logs below, 1st line shows the the GET request with the 301 code followed a few lines later but a GET with 200 success code

This is not a consistent happening as most requests are honored with a 200 success code.

crawler5138.ask.com - - [17/Nov/2009:01:58:03 -0500] "GET /robots.txt HTTP/1.1" 301 567 "-" "Mozilla/5.0

crawler5138.ask.com - - [17/Nov/2009:01:58:09 -0500] "GET /robots.txt HTTP/1.1" 200 5227 "-" "Mozilla/5.0