| Selective bot blocking
Scanning rules |
fabfurs
msg:1527279 | 4:09 am on Feb 16, 2004 (gmt 0) |
Do robots obey the wildcard USER-AGENT and then look for any specific instructions related to their UA, for instance:
USER-AGENT: *
Disallow:
Disallow: /dl/
Disallow: /features/ USER-AGENT: Googlebot
Disallow: /catalog/ Will Googlebot obey the first rule and not spider "dl", "features" and then read it's USER-AGENT list and not scan "catalog" or must "dl" and "features" be added?
|
ThomasB
msg:1527280 | 11:43 am on Feb 16, 2004 (gmt 0) |
From my experience it should work, at least for other bots. But I think GB should be able to avoid spidering /catalog/, too.
|
|
|
