Welcome to WebmasterWorld Guest from 107.20.5.156

Forum Moderators: goodroi

Message Too Old, No Replies

301 redirects & robots.txt

What is the implication of a 301 redirect on robots.txt

     

Jerry_Pannu

7:32 am on Feb 7, 2004 (gmt 0)

10+ Year Member



Robots.txt is very important from the SEO perspective. By default, the robot looks for the robots.txt file in the root of the original domain. The question is, what happens exactly when the spiders face a 301 redirect?
Does the robot skip it on not finding the robots.txt at the default location or goes on to the redirected landing domain as well to pick up indexing instrucions?

DaveAtIFG

10:17 pm on Feb 7, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I used a server header check program and a site that's well known to me. I received a 301 for requests to both example.com and example.com/robots.txt and 200s for requests to www.example.com and www.example.com/robots.txt, just as one would expect. :)
 

Featured Threads

Hot Threads This Week

Hot Threads This Month