Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- How to ban (compatible ; type requests


larryhatch - 9:05 pm on Jun 23, 2006 (gmt 0)


Thanks guys!

Wilderness: you wrote:
" I use a single line for these:
SetEnvIf User-Agent "compatible ;" keep_out ..

Do you put this line directly into .htaccess, or someplace else?

Are you sure I don't have to do something special (escape etc.) with the space?
I do NOT want to disallow compatible; with the semicolon properly placed of course.

If it goes into .htaccess, where exactly do I put it?
My present .htaccess looks like this:

RewriteEngine On

# By Referer:
RewriteCond %{HTTP_REFERER} forumxx\.#*$!xx\.xxx [NC,OR]
RewriteCond %{HTTP_REFERER} forumyy\.fok\.nl [NC,OR]

# By user agent:
RewriteCond %{HTTP_USER_AGENT} larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Java/1 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft\ URL\ Control [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Missigua [NC]
RewriteRule .* - [F]

- - - -

I just want a single foolproof line I can insert somewhere in the above. -Larry


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3309.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com