Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Selective bot blocking

Scanning rules

4:09 am on Feb 16, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 22, 2003
votes: 0

Do robots obey the wildcard USER-AGENT and then look for any specific instructions related to their UA, for instance:
Disallow: /dl/
Disallow: /features/

USER-AGENT: Googlebot
Disallow: /catalog/

Will Googlebot obey the first rule and not spider "dl", "features" and then read it's USER-AGENT list and not scan "catalog" or must "dl" and "features" be added?

11:43 am on Feb 16, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 5, 2002
votes: 0

From my experience it should work, at least for other bots. But I think GB should be able to avoid spidering /catalog/, too.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members