homepage Welcome to WebmasterWorld Guest from 50.17.27.205
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Where should I place robots.txt?
robots.txt location root directory or subdirectory with redirect 301
cssatsc




msg:3915819
 3:13 pm on May 18, 2009 (gmt 0)

I am just beginning to learn about robots.txt. I read the article about it in Wikipedia and a few other source but I am not completely sure where to place it:

On one hand every source says "place it in the root directory".

On the other hand, my site has a redirect 301 from <my-site>.com to <my-site>.com/shop.

So, should I place robots.txt in <my-site>.com? or <my-site>.com/shop?

(please note that at the moment (with the redirect 301) no one is able to see <my-site>.com or any subdirectory other than <my-site>.com/shop.)

Thanks.

 

bill




msg:3916194
 4:59 am on May 19, 2009 (gmt 0)

If your root directory is <my-site>.com/shop, then place robots.txt there.

phranque




msg:3920111
 8:17 am on May 26, 2009 (gmt 0)

the request will be for example.com/robots.txt, so you have two problems to solve.
the first is to make sure that request gets redirected to the correct resource.
the 2nd is to test and make sure the SE's will follow a redirect to robots.txt.

g1smd




msg:3920530
 7:21 pm on May 26, 2009 (gmt 0)

Ideally, your redirect to /shop/ should not happen if the request is for /robots.txt - you'll need to adjust your redirect rules for that.

cssatsc




msg:3920537
 7:46 pm on May 26, 2009 (gmt 0)

Well, I just checked with Google and will not accept any location other than <my-site>.com/robots.txt.

Fortunately, despite the redirect, accessing <my-site>.com/robots.txt directly is not a problem at all (given correct file access premissions, e.g. 644 on Linux) -Google was able to access it as well as my browser.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved