homepage Welcome to WebmasterWorld Guest from 54.166.122.65
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Multiple robots.txt files in different directories -will it work?
placing a robots.txt file in sub directories.
Michel Samuel

5+ Year Member



 
Msg#: 3472294 posted 12:18 am on Oct 9, 2007 (gmt 0)

I had an idea and I'm not certain if it would work or how to impliment it to test.

Suppose I'm thinking about only blocking 1 or 2 bots from my site. But only allow to access my domain root directory.

Therefore I create a robots.txt like this in the root.

user-agent 2 bots blocked
Disallow: /

But I then go to my sub directories and block them from all robots by placing a seperate robots.txt file in each of them.

User-agent: *
Disallow: /

Would this work?
Or is robots.txt only read if it is in the root directory of the domain?
Or are the bots instructed only to search for 1 inclusion of the file per domain?

Curiosity bothers me because nothing seems to cover this idea on any of the lititure listed on the net.

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3472294 posted 12:55 am on Oct 9, 2007 (gmt 0)

By definition, robots.txt is located in the root directory of a site, and applies to the entire site. robots.txt files in subdirectories will not be fetched, and can therefore have no effect.

Ref: A Standard for Robot Exclusion [robotstxt.org]

Jim

Michel Samuel

5+ Year Member



 
Msg#: 3472294 posted 7:44 pm on Oct 9, 2007 (gmt 0)

I think my way would work much better.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved