Page is a not externally linkable
wilderness - 10:00 pm on Dec 11, 2011 (gmt 0)
I just realized that most requests are direct and have no referrer. The referrer is blank. I used the following in my htaccess as of now and the spam seems to have completely stopped:
RewriteCond %{HTTP:Accept-Language} ^$ [OR]
RewriteCond %{HTTP_REFERER} ^$
RewriteRule .* - [F,L]
I know this will block quite a few legit users as well but is there any other solution to this? Also will this hinder search engine bots from crawling my site?
I also noticed that all these bots are requesting only one page on my site. This page is the largest page my site has. Is it possible to apply this htaccess rule to this single page alone and not the whole site?
Some of the most consistent UAs are as follows:
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; FunWebProducts)
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) w:PACBHO60
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; FunWebProducts; GTB7.0; SLCC2; .NET CLR 2.0.50727; .NET CL
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; BTRS28059; SearchToolbar 1.2; GTB7.2; SLCC2; .NET CLR 2.0.50727;
I am not exactly sure if referring pages and requested pages are the same, but I think they are.
PS: the last three UAs are truncated. I am not able to copy the full text for some reason.