Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- How to ban (compatible ; type requests


wilderness - 9:52 pm on Jun 23, 2006 (gmt 0)


RewriteEngine On

# By Referer:
RewriteCond %{HTTP_REFERER} forumxx\.#*$!xx\.#*$! [NC,OR]
RewriteCond %{HTTP_REFERER} forumyy\.fok\.nl [NC,OR]

# By user agent:
RewriteCond %{HTTP_USER_AGENT} larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Java/1 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft\ URL\ Control [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Missigua [NC]
RewriteRule .* - [F]

Larry if you use the additional options that I provided in Msg#9 of this thread?

Than your previously installed Rewrites would go below what I provided in your htaccess.

so it would now read:

Options -Indexes
<Limit GET>
SetEnvIf User-Agent "compatible ;" keep_out
order allow,deny
deny from 000.xx.#*$!.
deny from xx.xxx.xx.xxx
allow from all
deny from env=keep_out
</Limit>
RewriteEngine On
# By Referer:
RewriteCond %{HTTP_REFERER} forumxx\.#*$!xx\.xxx [NC,OR]
RewriteCond %{HTTP_REFERER} forumyy\.fok\.nl [NC,OR]
# By user agent:
RewriteCond %{HTTP_USER_AGENT} larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Java/1 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft\ URL\ Control [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Missigua [NC]
RewriteRule .* - [F]

edited by wilderness.

In edition you may combine your UA's in your example (three to a single line)using the "or pipe charcter"
RewriteCond %{HTTP_USER_AGENT} larbin¦Java/1¦HTTrack [NC,OR]

BTW, don't forget that the forum breaks the pipe chracter into a split line and these MUST be corrected.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3309.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com