Welcome to WebmasterWorld Guest from 107.20.110.201

Forum Moderators: goodroi

Message Too Old, No Replies

Where should I place robots.txt?

robots.txt location root directory or subdirectory with redirect 301

     

cssatsc

3:13 pm on May 18, 2009 (gmt 0)

5+ Year Member



I am just beginning to learn about robots.txt. I read the article about it in Wikipedia and a few other source but I am not completely sure where to place it:

On one hand every source says "place it in the root directory".

On the other hand, my site has a redirect 301 from <my-site>.com to <my-site>.com/shop.

So, should I place robots.txt in <my-site>.com? or <my-site>.com/shop?

(please note that at the moment (with the redirect 301) no one is able to see <my-site>.com or any subdirectory other than <my-site>.com/shop.)

Thanks.

bill

4:59 am on May 19, 2009 (gmt 0)

WebmasterWorld Administrator bill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



If your root directory is <my-site>.com/shop, then place robots.txt there.

phranque

8:17 am on May 26, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



the request will be for example.com/robots.txt, so you have two problems to solve.
the first is to make sure that request gets redirected to the correct resource.
the 2nd is to test and make sure the SE's will follow a redirect to robots.txt.

g1smd

7:21 pm on May 26, 2009 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Ideally, your redirect to /shop/ should not happen if the request is for /robots.txt - you'll need to adjust your redirect rules for that.

cssatsc

7:46 pm on May 26, 2009 (gmt 0)

5+ Year Member



Well, I just checked with Google and will not accept any location other than <my-site>.com/robots.txt.

Fortunately, despite the redirect, accessing <my-site>.com/robots.txt directly is not a problem at all (given correct file access premissions, e.g. 644 on Linux) -Google was able to access it as well as my browser.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month