Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Selective bot blocking

Scanning rules



4:09 am on Feb 16, 2004 (gmt 0)

10+ Year Member

Do robots obey the wildcard USER-AGENT and then look for any specific instructions related to their UA, for instance:
Disallow: /dl/
Disallow: /features/

USER-AGENT: Googlebot
Disallow: /catalog/

Will Googlebot obey the first rule and not spider "dl", "features" and then read it's USER-AGENT list and not scan "catalog" or must "dl" and "features" be added?


11:43 am on Feb 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

From my experience it should work, at least for other bots. But I think GB should be able to avoid spidering /catalog/, too.

Featured Threads

Hot Threads This Week

Hot Threads This Month