homepage Welcome to WebmasterWorld Guest from 54.204.73.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Putting "disallow: /" with listing subdirectories?
Would the first line would override the rest?
pocket723

5+ Year Member



 
Msg#: 3870625 posted 5:45 am on Mar 15, 2009 (gmt 0)

Sometimes I see sites with the following robots.txt file:

User-agent: *
Disallow: /
Disallow: /private/
Disallow: /user/

Does the line "Disallow: /" override the rest and block crawlers from accessing the entire site? Or does listing specific subdirectories override the top line and enable crawlers to access the rest of the site?

 

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3870625 posted 6:59 am on Mar 16, 2009 (gmt 0)

welcome to WebmasterWorld [webmasterworld.com], pocket723!

the disallow syntax of the robots exclusion protocol matches left-to-right.
in your example the 2nd and 3rd disallow are redundant and it is disallowing everything - including, in and under the root directory.

the default is to allow everything or if you want to whitelist a specific bot and disallow all others from those two directories:
User-agent: specific-bot
Disallow:

User-agent: *
Disallow: /private/
Disallow: /user/

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved