Welcome to WebmasterWorld Guest from 54.198.206.232

Forum Moderators: goodroi

Message Too Old, No Replies

Putting "disallow: /" with listing subdirectories?

Would the first line would override the rest?

     
5:45 am on Mar 15, 2009 (gmt 0)

New User

5+ Year Member

joined:Mar 15, 2009
posts: 1
votes: 0


Sometimes I see sites with the following robots.txt file:

User-agent: *
Disallow: /
Disallow: /private/
Disallow: /user/

Does the line "Disallow: /" override the rest and block crawlers from accessing the entire site? Or does listing specific subdirectories override the top line and enable crawlers to access the rest of the site?

6:59 am on Mar 16, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10553
votes: 13


welcome to WebmasterWorld [webmasterworld.com], pocket723!

the disallow syntax of the robots exclusion protocol matches left-to-right.
in your example the 2nd and 3rd disallow are redundant and it is disallowing everything - including, in and under the root directory.

the default is to allow everything or if you want to whitelist a specific bot and disallow all others from those two directories:
User-agent: specific-bot
Disallow:

User-agent: *
Disallow: /private/
Disallow: /user/