| Welcome to WebmasterWorld Guest from 184.108.40.206 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Multiple robots.txt files in different directories -will it work?|
placing a robots.txt file in sub directories.
| 12:18 am on Oct 9, 2007 (gmt 0)|
I had an idea and I'm not certain if it would work or how to impliment it to test.
Suppose I'm thinking about only blocking 1 or 2 bots from my site. But only allow to access my domain root directory.
Therefore I create a robots.txt like this in the root.
user-agent 2 bots blocked
But I then go to my sub directories and block them from all robots by placing a seperate robots.txt file in each of them.
Would this work?
Or is robots.txt only read if it is in the root directory of the domain?
Or are the bots instructed only to search for 1 inclusion of the file per domain?
Curiosity bothers me because nothing seems to cover this idea on any of the lititure listed on the net.
| 12:55 am on Oct 9, 2007 (gmt 0)|
By definition, robots.txt is located in the root directory of a site, and applies to the entire site. robots.txt files in subdirectories will not be fetched, and can therefore have no effect.
Ref: A Standard for Robot Exclusion [robotstxt.org]
| 7:44 pm on Oct 9, 2007 (gmt 0)|
I think my way would work much better.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved