Welcome to WebmasterWorld Guest from 54.167.0.111

Forum Moderators: goodroi

Message Too Old, No Replies

Putting "disallow: /" with listing subdirectories?

Would the first line would override the rest?

     

pocket723

5:45 am on Mar 15, 2009 (gmt 0)

5+ Year Member



Sometimes I see sites with the following robots.txt file:

User-agent: *
Disallow: /
Disallow: /private/
Disallow: /user/

Does the line "Disallow: /" override the rest and block crawlers from accessing the entire site? Or does listing specific subdirectories override the top line and enable crawlers to access the rest of the site?

phranque

6:59 am on Mar 16, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



welcome to WebmasterWorld [webmasterworld.com], pocket723!

the disallow syntax of the robots exclusion protocol matches left-to-right.
in your example the 2nd and 3rd disallow are redundant and it is disallowing everything - including, in and under the root directory.

the default is to allow everything or if you want to whitelist a specific bot and disallow all others from those two directories:
User-agent: specific-bot
Disallow:

User-agent: *
Disallow: /private/
Disallow: /user/

 

Featured Threads

Hot Threads This Week

Hot Threads This Month