Welcome to WebmasterWorld Guest from 54.144.231.243

Forum Moderators: goodroi

Putting "disallow: /" with listing subdirectories?

Would the first line would override the rest?

   
5:45 am on Mar 15, 2009 (gmt 0)

5+ Year Member



Sometimes I see sites with the following robots.txt file:

User-agent: *
Disallow: /
Disallow: /private/
Disallow: /user/

Does the line "Disallow: /" override the rest and block crawlers from accessing the entire site? Or does listing specific subdirectories override the top line and enable crawlers to access the rest of the site?

6:59 am on Mar 16, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



welcome to WebmasterWorld [webmasterworld.com], pocket723!

the disallow syntax of the robots exclusion protocol matches left-to-right.
in your example the 2nd and 3rd disallow are redundant and it is disallowing everything - including, in and under the root directory.

the default is to allow everything or if you want to whitelist a specific bot and disallow all others from those two directories:
User-agent: specific-bot
Disallow:

User-agent: *
Disallow: /private/
Disallow: /user/

 

Featured Threads

Hot Threads This Week

Hot Threads This Month