Welcome to WebmasterWorld Guest from 54.145.104.199

Forum Moderators: goodroi

Message Too Old, No Replies

Multiple robots.txt files in different directories -will it work?

placing a robots.txt file in sub directories.

     
12:18 am on Oct 9, 2007 (gmt 0)

5+ Year Member



I had an idea and I'm not certain if it would work or how to impliment it to test.

Suppose I'm thinking about only blocking 1 or 2 bots from my site. But only allow to access my domain root directory.

Therefore I create a robots.txt like this in the root.

user-agent 2 bots blocked
Disallow: /

But I then go to my sub directories and block them from all robots by placing a seperate robots.txt file in each of them.

User-agent: *
Disallow: /

Would this work?
Or is robots.txt only read if it is in the root directory of the domain?
Or are the bots instructed only to search for 1 inclusion of the file per domain?

Curiosity bothers me because nothing seems to cover this idea on any of the lititure listed on the net.

12:55 am on Oct 9, 2007 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



By definition, robots.txt is located in the root directory of a site, and applies to the entire site. robots.txt files in subdirectories will not be fetched, and can therefore have no effect.

Ref: A Standard for Robot Exclusion [robotstxt.org]

Jim

7:44 pm on Oct 9, 2007 (gmt 0)

5+ Year Member



I think my way would work much better.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month