Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Thousands of Spambot IPs Hitting my Site


wilderness - 10:00 pm on Dec 11, 2011 (gmt 0)


I just realized that most requests are direct and have no referrer. The referrer is blank. I used the following in my htaccess as of now and the spam seems to have completely stopped:

RewriteCond %{HTTP:Accept-Language} ^$ [OR]
RewriteCond %{HTTP_REFERER} ^$
RewriteRule .* - [F,L]

I know this will block quite a few legit users as well but is there any other solution to this? Also will this hinder search engine bots from crawling my site?

I also noticed that all these bots are requesting only one page on my site. This page is the largest page my site has. Is it possible to apply this htaccess rule to this single page alone and not the whole site?

Some of the most consistent UAs are as follows:

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; FunWebProducts)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) w:PACBHO60

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; FunWebProducts; GTB7.0; SLCC2; .NET CLR 2.0.50727; .NET CL

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; BTRS28059; SearchToolbar 1.2; GTB7.2; SLCC2; .NET CLR 2.0.50727;

I am not exactly sure if referring pages and requested pages are the same, but I think they are.

PS: the last three UAs are truncated. I am not able to copy the full text for some reason.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4396669.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com