Asia_Expat - 7:04 pm on Apr 1, 2008 (gmt 0)

I recently noticed a peaking in my server load. At the same time, I noticed on my forum a certain range of IP addresses continually crawling over and over, they seemed particularly interested in the 'register' page.

I did some digging around about this IP range and traced them to a Swedish company called Munax. They offer the following statement about their spidering activities, in which they openly claim to cloak their spiders to appear as a regular human being...
[munax.com...]

They also claim to respect the robots.txt file but I can assure you that claim is false. My forum is carefully managed in my robots file and these IP's are totally ignoring it. Further, they are crawling so aggressively and with such repetition, it's bringing my server to it's knees at times and also making it look like there are many more visitors than there actually is.

The IP range, as far as I can tell, is...
82.99.30.0 - 82.99.30.127

What can I do about this? If I place the following in my htaccess file in the forum directory, will it work and will I potentially be doing any harm?...

<Files *>
order allow,deny
allow from all
deny from 82.99.30.0/127
</Files>