Do robots obey the wildcard USER-AGENT and then look for any specific instructions related to their UA, for instance: USER-AGENT: * Disallow: Disallow: /dl/ Disallow: /features/
USER-AGENT: Googlebot Disallow: /catalog/
Will Googlebot obey the first rule and not spider "dl", "features" and then read it's USER-AGENT list and not scan "catalog" or must "dl" and "features" be added?
ThomasB
11:43 am on Feb 16, 2004 (gmt 0)
From my experience it should work, at least for other bots. But I think GB should be able to avoid spidering /catalog/, too.
