homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Where should I place robots.txt?
robots.txt location root directory or subdirectory with redirect 301

 3:13 pm on May 18, 2009 (gmt 0)

I am just beginning to learn about robots.txt. I read the article about it in Wikipedia and a few other source but I am not completely sure where to place it:

On one hand every source says "place it in the root directory".

On the other hand, my site has a redirect 301 from <my-site>.com to <my-site>.com/shop.

So, should I place robots.txt in <my-site>.com? or <my-site>.com/shop?

(please note that at the moment (with the redirect 301) no one is able to see <my-site>.com or any subdirectory other than <my-site>.com/shop.)




 4:59 am on May 19, 2009 (gmt 0)

If your root directory is <my-site>.com/shop, then place robots.txt there.


 8:17 am on May 26, 2009 (gmt 0)

the request will be for example.com/robots.txt, so you have two problems to solve.
the first is to make sure that request gets redirected to the correct resource.
the 2nd is to test and make sure the SE's will follow a redirect to robots.txt.


 7:21 pm on May 26, 2009 (gmt 0)

Ideally, your redirect to /shop/ should not happen if the request is for /robots.txt - you'll need to adjust your redirect rules for that.


 7:46 pm on May 26, 2009 (gmt 0)

Well, I just checked with Google and will not accept any location other than <my-site>.com/robots.txt.

Fortunately, despite the redirect, accessing <my-site>.com/robots.txt directly is not a problem at all (given correct file access premissions, e.g. 644 on Linux) -Google was able to access it as well as my browser.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved