homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Gold Sponsor 2015!
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

What am I doing wrong guys to block IP,s
rDNSbot help re - IP,s

5+ Year Member

Msg#: 3409508 posted 1:55 pm on Jul 31, 2007 (gmt 0)

Gang I need help...I recently followed a link from Incredibills.blogspot,and found a great couple of threads talking about a subject,which had aroused my suspicions in the hosting back office..regarding site hijacking and fake robots and scrapers.
Thanks to the guys of those threads i organised my .htaccess files and.. surprise my my real traffic and views have been steadily on the rise.

However guys after putting in this code below in .htaccess. from those particular threads..
It works great..but for some reasons i cannot block individual IP's.
The ip,s I want to block and put in .htacces still continue to crawl my site..what can i do to rectify this?

many thanks

here is exactley my .htaccess
AddHandler application/x-httpd-php5 .php
<Files 403.shtml>
order allow,deny
allow from all
<FilesMatch "\.(s?html?Šphp[45]?)$">
BrowserMatchNoCase Googlebot rDNSbot
BrowserMatchNoCase msnbot rDNSbot
BrowserMatchNoCase Slurp rDNSbot
BrowserMatchNoCase Teoma rDNSbot
SetEnvIf Request_URI "/path-to-custom-403-page\.shtml$" AllowAll
Order Deny,Allow
Deny from env=rDNSbot
Allow from env=AllowAll
Allow from googlebot.com
Allow from search.live.com
Allow from crawl.yahoo.net
Allow from ask.com
Allow from inktomisearch.com

deny from 38.99.44.**
deny from 72.36.94.***
deny from 128.174.254.**
deny from 216.95.221.**
deny from 213.232.196.***
deny from 81.172.95.***
deny from 66.249.17.***

[edited by: volatilegx at 1:53 pm (utc) on Aug. 2, 2007]
[edit reason] obfuscated IP addresses [/edit]



WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3409508 posted 2:05 pm on Aug 2, 2007 (gmt 0)

answered in Apache Web Server.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved