Thousands of Spambot IPs Hitting my Site Eating up Bandwidth

Thousands of Spambot IPs Hitting my Site

[spiritualseo]

Hey guys, need someone's help here. From the past week or so the bandwidth usage of my site increased from 1GB a month to 12GB a Day!

Awstats indicates that there are a range of unique IPs hitting my site and requesting thousands of pages with every visit. Most of these IPs seem to be originating from within the US which is funny. I have blocked China, Brazil and some other countries in my HTaccess but the hits continue.

Here are some example IPs that hit my site as of now and requested thousands of pages.

24.181.178.3
216.6.134.27
70.182.254.242
99.98.188.110
75.134.95.208

Can anyone please explain what is happening? And what can I possibly do to stop this? If this continues, my site will go offline within a week or so.

[spiritualseo]

Please take a look at these ips, these are just a few from thousands that hit my site almost every second. And each one requests around 1000 pages. My site has only around 100 pages so perhaps they request a page over and over again:

65.35.111.110
71.197.69.88
124.123.51.38
173.198.98.134
198.138.135.123
24.159.55.211
50.40.131.171
174.16.100.103
74.131.129.17
75.94.108.222
98.218.136.190
69.244.107.77
68.185.252.101

The funny thing is that all of them look unique and all of them have a verified DNS. How can this be? Could someone please answer.

[wilderness]

This is better addressed in Search Engine Spider and User Identification Forum [webmasterworld.com]

[wilderness]

FWIW,
Awstats and these IP's (at least alone) are useless.

You need to view your "raw visitor logs" and see if there is some correlation between the the visitors User Agents and the referring page (s).

[spiritualseo]

Thanks, the referring pages are generally pages within my own site. Have posted it in the search engine spider thread, but yet to get approved.

[wilderness]

the referring pages are generally pages within my own site

1) You haven't mentioned the User Agents?
Is there any consistency there?
Perhaps obvious bot names or badly formed browser UA's?

2) are the referring pages the same as the requested pages?

[spiritualseo]

I just realized that most requests are direct and have no referrer. The referrer is blank. I used the following in my htaccess as of now and the spam seems to have completely stopped:

RewriteCond %{HTTP:Accept-Language} ^$ [OR]
RewriteCond %{HTTP_REFERER} ^$
RewriteRule .* - [F,L]

I know this will block quite a few legit users as well but is there any other solution to this? Also will this hinder search engine bots from crawling my site?

I also noticed that all these bots are requesting only one page on my site. This page is the largest page my site has. Is it possible to apply this htaccess rule to this single page alone and not the whole site?

Some of the most consistent UAs are as follows:

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; FunWebProducts)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) w:PACBHO60

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; FunWebProducts; GTB7.0; SLCC2; .NET CLR 2.0.50727; .NET CL

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; BTRS28059; SearchToolbar 1.2; GTB7.2; SLCC2; .NET CLR 2.0.50727;

I am not exactly sure if referring pages and requested pages are the same, but I think they are.

PS: the last three UAs are truncated. I am not able to copy the full text for some reason.

[Al_McCann]

In the day job as the internet police, I have seen this behavior in our content filter logs for one specific website run by a local church. When a user visited that website in Internet Explorer, it got stuck in some kind of redirect loop, and requested the same error page almost a half million times in a little under two hours.

When I visit that page in Firefox, I got a "the way the page redirects will never complete" error.

Form where I sit, this looks like some kind of bug in Internet Explorer combined with a broken redirect. Your user example agents are all IE.

I also see "PACBHO60", "FunWebProducts" and "SearchToolbar" in your examples, they seem to be spyware/malware , maybe they are also interfering in how IE redirects.

Al

[Pfui]

Al, welcome. This thread sort of duped/looped itself into the Search Engine Spider and User Agent Identification Forum where it continues... [webmasterworld.com...] Here's hoping more info = more ideas:)