Forum Moderators: goodroi

Message Too Old, No Replies

can you have 2 robots.txt files on site?

one in main and one in subdomain

         

stiksandstones

5:09 pm on Aug 4, 2005 (gmt 0)

10+ Year Member



I have an estore located at my site
www.mysite.com
and my store is on a subdomain
[store.mysite.com...]

I was looking at logs (disregard dates, its the same no matter the date) and saw this;;

I have my robots.txt file in main root of my site. I was checking my logs and find it says
66.249.64.52 - - [01/Aug/2005:23:17:00 -0400] "GET /robots.txt HTTP/1.0" 200 632 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66.249.64.52 - - [01/Aug/2005:23:17:00 -0400] "GET / HTTP/1.0" 200 6397 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

I know the 200 code means it was a success
but nothing more, isnt it supposed to crawl ALL my pages? looks like it didnt crawl any.


So then I checked my logs for my subdomain (store.mysite.com) and found...

66.249.64.16 - - [28/Mar/2005:21:38:50 -0800] "GET /robots.txt HTTP/1.0" 404 - "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
66.249.64.16 - - [28/Mar/2005:21:38:57 -0800] "GET / HTTP/1.0" 200 14980 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

404 means error, didnt find the robots.txt in my [store.mysite.com...] Am I supposed to have a robots in my subdomain AND my main domain?

ThomasB

6:11 pm on Aug 4, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You certainly need a robots.txt file for any (sub-)domain where you want robots to obey it.

Jioffri

2:24 am on Aug 14, 2005 (gmt 0)

10+ Year Member



But shouldn't the robots.txt in the root take precedence over the entire domain as long as all subdomains are based on the same TLD domain name? For example, if the website is [anywebsite.com...] and the estore subdomain is [estore.anywebsite.com?...]

sokybiz

4:11 pm on Aug 14, 2005 (gmt 0)

10+ Year Member



I'm thinking that you should have a robots.txt file in the root of your subdomain. But... not knowing your configuration all I can offer is an easy way to answer your question.

Go to [store.mysite.com...] ... if not found then the answer would be yes... you need a separate file.

Lord Majestic

1:59 pm on Aug 17, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



But shouldn't the robots.txt in the root take precedence over the entire domain as long as all subdomains are based on the same TLD domain name?

No - one robots.txt per unique domain name (including subdomains), per protocol and per port.

Jioffri

5:56 am on Aug 18, 2005 (gmt 0)

10+ Year Member



"No - one robots.txt per unique domain name (including subdomains), per protocol and per port."
--------------------

Maj,

OK, thanks for clarifying that. However, should I ask my hosting service to add and edit the robots.txt file for me since I do not have one on my webstore which is a subdomain of my tld? Or is there another way to add one or perhaps resource(s) you can refer me to that will explain how to do this?

Jio

stiksandstones

6:24 am on Aug 18, 2005 (gmt 0)

10+ Year Member



I put a robots in my main root
and in my subdomain...

What is the risk I run of having 2 now?

Lord Majestic

12:10 am on Aug 19, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



There is no risk.