Welcome to WebmasterWorld Guest from 54.227.52.24

Forum Moderators: goodroi

Message Too Old, No Replies

Selective bot blocking

Scanning rules

     

fabfurs

4:09 am on Feb 16, 2004 (gmt 0)

10+ Year Member



Do robots obey the wildcard USER-AGENT and then look for any specific instructions related to their UA, for instance:
USER-AGENT: *
Disallow:
Disallow: /dl/
Disallow: /features/

USER-AGENT: Googlebot
Disallow: /catalog/

Will Googlebot obey the first rule and not spider "dl", "features" and then read it's USER-AGENT list and not scan "catalog" or must "dl" and "features" be added?

ThomasB

11:43 am on Feb 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



From my experience it should work, at least for other bots. But I think GB should be able to avoid spidering /catalog/, too.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month