homepage Welcome to WebmasterWorld Guest from 184.73.40.21
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Where should I place robots.txt?
robots.txt location root directory or subdirectory with redirect 301
cssatsc

5+ Year Member



 
Msg#: 3915817 posted 3:13 pm on May 18, 2009 (gmt 0)

I am just beginning to learn about robots.txt. I read the article about it in Wikipedia and a few other source but I am not completely sure where to place it:

On one hand every source says "place it in the root directory".

On the other hand, my site has a redirect 301 from <my-site>.com to <my-site>.com/shop.

So, should I place robots.txt in <my-site>.com? or <my-site>.com/shop?

(please note that at the moment (with the redirect 301) no one is able to see <my-site>.com or any subdirectory other than <my-site>.com/shop.)

Thanks.

 

bill

WebmasterWorld Administrator bill us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3915817 posted 4:59 am on May 19, 2009 (gmt 0)

If your root directory is <my-site>.com/shop, then place robots.txt there.

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3915817 posted 8:17 am on May 26, 2009 (gmt 0)

the request will be for example.com/robots.txt, so you have two problems to solve.
the first is to make sure that request gets redirected to the correct resource.
the 2nd is to test and make sure the SE's will follow a redirect to robots.txt.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3915817 posted 7:21 pm on May 26, 2009 (gmt 0)

Ideally, your redirect to /shop/ should not happen if the request is for /robots.txt - you'll need to adjust your redirect rules for that.

cssatsc

5+ Year Member



 
Msg#: 3915817 posted 7:46 pm on May 26, 2009 (gmt 0)

Well, I just checked with Google and will not accept any location other than <my-site>.com/robots.txt.

Fortunately, despite the redirect, accessing <my-site>.com/robots.txt directly is not a problem at all (given correct file access premissions, e.g. 644 on Linux) -Google was able to access it as well as my browser.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved