homepage Welcome to WebmasterWorld Guest from 54.83.133.189
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
301 redirects & robots.txt
What is the implication of a 301 redirect on robots.txt
Jerry_Pannu




msg:1525513
 7:32 am on Feb 7, 2004 (gmt 0)

Robots.txt is very important from the SEO perspective. By default, the robot looks for the robots.txt file in the root of the original domain. The question is, what happens exactly when the spiders face a 301 redirect?
Does the robot skip it on not finding the robots.txt at the default location or goes on to the redirected landing domain as well to pick up indexing instrucions?

 

DaveAtIFG




msg:1525514
 10:17 pm on Feb 7, 2004 (gmt 0)

I used a server header check program and a site that's well known to me. I received a 301 for requests to both example.com and example.com/robots.txt and 200s for requests to www.example.com and www.example.com/robots.txt, just as one would expect. :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved