Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt - subdomains and domain

where to place

         

Rose_1171

7:59 am on Feb 17, 2006 (gmt 0)

10+ Year Member



Im sure this question has been asked hundreds of time! Anyway here i go again
I have three subdomains and one main domain.

Do I need to place robots.txt in all these folders( subdomain 1 htdocs, subdomain2 htdocs.. and htdocs (main root folder)?
or just one main htdocs (root)

Thanks
Rosie

Dijkgraaf

9:02 am on Feb 17, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi Rosie

If they point to different folders, then yes, you would need to place a robots.txt in the root folder for each (sub)domain if you want to place any restrictions on visiting bots. All disallow statements would be from the root folder of that domain.
The robots.txt for each particular one should be viewable if you were to go
http://www.example.com/robots.txt
[subdomain.example.com...]

Also some people have their sub domains pointing to a sub folder of the main domain, so that a request for
www.example.com/subdomainfolder/examplepage.htm
and
subdomain.example.com/examplepage.htm
would serve up the same page.
In that instance I would also place a disallow for the subdomainfolder in the main domains robots.txt to avoid problems.

incrediBILL

8:06 am on Feb 18, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



It doesn't really work anymore except for legit bots only and even then Google peeks.

See my comments here:
[webmasterworld.com...]