Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Microsoft or not?

keyplyr - 1:12 am on Jan 5, 2013 (gmt 0)


I've blocked blank UA for close to 10 years:

RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REQUEST_METHOD} !^HEAD$
RewriteRule !^(favicon\.ico|403\.html|w3c/p3p\.xml|w3c/policy1\.xml|robots\.txt)$ - [F]

But allow HEAD requests for the above files.

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4533172.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com