Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- deny all


wilderness - 2:51 am on May 12, 2012 (gmt 0)


Thanks wilderness. You can use a Limit container. If you want to use mod_rewrite instead, you could use:

RewriteCond %{ENV:dont_allow} ^1$
RewriteRule .* - [F]


Key_Master

How would I add the following lines into the ENV?

RewriteCond %{REMOTE_ADDR} !^66\.249\.(6[4-9]|[78][0-9]|9[0-5])\.
RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteRule .* - [F]
RewriteCond %{REMOTE_ADDR} ^65\.5[2-5]\. [OR]
RewriteCond %{REMOTE_ADDR} ^70\.37\. [OR]
RewriteCond %{REMOTE_ADDR} ^157\.[45][0-9]\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.[67][0-9]\.
RewriteCond %{HTTP_USER_AGENT} !(Bingbot|msnbot) [NC]
RewriteRule !^robots\.txt$ - [F]


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4422323.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com