homepage Welcome to WebmasterWorld Guest from 107.20.109.52
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Multiple robots.txt files in different directories -will it work?
placing a robots.txt file in sub directories.
Michel Samuel




msg:3472296
 12:18 am on Oct 9, 2007 (gmt 0)

I had an idea and I'm not certain if it would work or how to impliment it to test.

Suppose I'm thinking about only blocking 1 or 2 bots from my site. But only allow to access my domain root directory.

Therefore I create a robots.txt like this in the root.

user-agent 2 bots blocked
Disallow: /

But I then go to my sub directories and block them from all robots by placing a seperate robots.txt file in each of them.

User-agent: *
Disallow: /

Would this work?
Or is robots.txt only read if it is in the root directory of the domain?
Or are the bots instructed only to search for 1 inclusion of the file per domain?

Curiosity bothers me because nothing seems to cover this idea on any of the lititure listed on the net.

 

jdMorgan




msg:3472316
 12:55 am on Oct 9, 2007 (gmt 0)

By definition, robots.txt is located in the root directory of a site, and applies to the entire site. robots.txt files in subdirectories will not be fetched, and can therefore have no effect.

Ref: A Standard for Robot Exclusion [robotstxt.org]

Jim

Michel Samuel




msg:3473048
 7:44 pm on Oct 9, 2007 (gmt 0)

I think my way would work much better.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved