Welcome to WebmasterWorld Guest from 54.198.118.102

Forum Moderators: goodroi

Message Too Old, No Replies

Selective bot blocking

Scanning rules

     
4:09 am on Feb 16, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 22, 2003
posts:116
votes: 0


Do robots obey the wildcard USER-AGENT and then look for any specific instructions related to their UA, for instance:
USER-AGENT: *
Disallow:
Disallow: /dl/
Disallow: /features/

USER-AGENT: Googlebot
Disallow: /catalog/

Will Googlebot obey the first rule and not spider "dl", "features" and then read it's USER-AGENT list and not scan "catalog" or must "dl" and "features" be added?

11:43 am on Feb 16, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 5, 2002
posts:1562
votes: 0


From my experience it should work, at least for other bots. But I think GB should be able to avoid spidering /catalog/, too.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members