Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- deny all


Key_Master - 12:07 am on Feb 28, 2012 (gmt 0)


Thanks wilderness. You can use a Limit container. If you want to use mod_rewrite instead, you could use:

RewriteCond %{ENV:dont_allow} ^1$
RewriteRule .* - [F]


With a little tweeking, deny,allow will work:

SetEnvIf Remote_Addr ^(?:66\.249\.(?:6[4-9]|[78][0-9]|9[0-5])|74\.125)\. allow

<Limit GET POST>
Order Deny,Allow
Deny from all
Allow from env=allow
</Limit>


Then make a white list of SetEnvIf conditions to allow what you want (using "allow" instead of "dont_allow"). You wouldn't need a "Deny all" list if that is the way you want to go.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4422323.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com