Welcome to WebmasterWorld Guest from 50.17.117.221

Forum Moderators: goodroi

Message Too Old, No Replies

Where should I place robots.txt?

robots.txt location root directory or subdirectory with redirect 301

     
3:13 pm on May 18, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:Apr 16, 2009
posts: 43
votes: 0


I am just beginning to learn about robots.txt. I read the article about it in Wikipedia and a few other source but I am not completely sure where to place it:

On one hand every source says "place it in the root directory".

On the other hand, my site has a redirect 301 from <my-site>.com to <my-site>.com/shop.

So, should I place robots.txt in <my-site>.com? or <my-site>.com/shop?

(please note that at the moment (with the redirect 301) no one is able to see <my-site>.com or any subdirectory other than <my-site>.com/shop.)

Thanks.

4:59 am on May 19, 2009 (gmt 0)

Administrator from JP 

WebmasterWorld Administrator bill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 12, 2000
posts:14480
votes: 49


If your root directory is <my-site>.com/shop, then place robots.txt there.
8:17 am on May 26, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


the request will be for example.com/robots.txt, so you have two problems to solve.
the first is to make sure that request gets redirected to the correct resource.
the 2nd is to test and make sure the SE's will follow a redirect to robots.txt.
7:21 pm on May 26, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Ideally, your redirect to /shop/ should not happen if the request is for /robots.txt - you'll need to adjust your redirect rules for that.
7:46 pm on May 26, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:Apr 16, 2009
posts: 43
votes: 0


Well, I just checked with Google and will not accept any location other than <my-site>.com/robots.txt.

Fortunately, despite the redirect, accessing <my-site>.com/robots.txt directly is not a problem at all (given correct file access premissions, e.g. 644 on Linux) -Google was able to access it as well as my browser.