Ddos attack - what to do

[doc_z]

I'm having a Ddos attack. It started 3 days ago and is still continuing.

So far more than 1 billion hits. There are peaks with 1 million hits per minute. The attacks are coming from thousands of IPs.

Any advice is welcome.

[wilderness]

Is there anything in common in the browser user agents?

[doc_z]

There are different user agents. However, there are a lot of unsual agents which appear often like "Mozilla/3.0 (Windows NT 5.0; rv:23.0) Gecko/20100101 Firefox/23.0"

[Leosghost]

What does your hosting company think..? at that many hits most would have switched your site off , or demanded your 1st born against bandwidth..

[not2easy]

I would have moved my site by this time if the host lets that continue for 3 days. Have you notified your host and asked them to take steps to stop it? If they can't or won't, it is time to move.
No one can use your site under those conditions, so keeping it active is only providing a target.

[brotherhood of LAN]

Some hosts will offer DDOS protection as part of their service. Agree that good hosts would typically take action as it's likely affecting other users on their network, usually by null routing your IP for 24 hours.

One advantage of using CDN's is you 'hide' your actual server IP holding the content. Some CDN providers also provide DDOS protection and methods to reactively block malicious traffic.

[lucy24]

Edit: Oops, buncha replies came in while I had the tab open. Scroll back about half an hour.

Is it your own server? Apache or IIS? Can you set up a firewall?

:: wait, stop, rewind ::

Mozilla/3.0

Mozilla THREE? Please say they're not getting anything but 403s, regardless of source. Now, some robots do go away faster if instead you send them off to contemplate their navels at either 127.0.0.1 or their own originating IP; others may give up if you start serving 404s (yes, you can do it manually even if the page actually exists). It may be worth experimenting.

Also look at the headers. See if there's anything present, missing or unusual compared to normal humans and well-behaved search engines.

But it's hard even to begin making suggestions if we don't know your relationship to the server. It's possible you have talked about it in other threads, but with issues like this it's best to give all information every time.

[aristotle]

Lucy is right -- It would be easier for people to help you if you would post some full entries from your logs, especially a series of 3 or 4 successive entries. In other words, Show everything

[doc_z]

Of course, first step was to ask my hosting company. (The website is just a small project and I don't even have my own server but managed hosting (Apache).)

They tried several things like limiting the number of request by IP and blocking IPs. However, this doesn't help because it wasn't enough filtering. Yesterday I counted 665 million hits and even if you block most of them, the rest is still too much. It isn't easy to block access because of the high number of different IPs and different user-agents.

My hosting company was also offering a professional solution but the price was very high (the price of a new car).

Indeed, a lot (40-50%) of the user-agents were Mozilla 3. They were getting 499 or 503. However, even this is making trouble if you're getting up to 1 million requests per minute.

Here are two examples from my log files, but I don't think that it's helpful:

31.176.221.198 - - [10/Dec/2014:09:08:27 +0100] "GET / HTTP/1.1" 503 608 "-" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.6; Trident/5.0)" "my-domain.com"
31.176.221.198 - - [10/Dec/2014:09:08:27 +0100] "GET / HTTP/1.1" 503 608 "-" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.6; Trident/5.0)" "my-domain.com"
31.176.221.198 - - [10/Dec/2014:09:08:27 +0100] "GET / HTTP/1.1" 503 608 "-" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.6; Trident/5.0)" "my-domain.com"
31.176.221.198 - - [10/Dec/2014:09:08:27 +0100] "GET / HTTP/1.1" 503 608 "-" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.6; Trident/5.0)" "my-domain.com"
31.176.221.198 - - [10/Dec/2014:09:08:27 +0100] "GET / HTTP/1.1" 503 608 "-" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.6; Trident/5.0)" "my-domain.com"

49.150.211.165 - - [10/Dec/2014:10:07:27 +0100] "GET / HTTP/1.1" 503 206 "-" "Mozilla/3.0 (Windows NT 5.1; rv:18.0) Gecko/20100101 Firefox/18.0" "my-domain.com"
49.150.211.165 - - [10/Dec/2014:10:07:27 +0100] "GET / HTTP/1.1" 503 206 "-" "Mozilla/3.0 (Windows NT 5.1; rv:18.0) Gecko/20100101 Firefox/18.0" "my-domain.com"
49.150.211.165 - - [10/Dec/2014:10:07:27 +0100] "GET / HTTP/1.1" 503 206 "-" "Mozilla/3.0 (Windows NT 5.1; rv:18.0) Gecko/20100101 Firefox/18.0" "my-domain.com"
49.150.211.165 - - [10/Dec/2014:10:07:27 +0100] "GET / HTTP/1.1" 503 206 "-" "Mozilla/3.0 (Windows NT 5.1; rv:18.0) Gecko/20100101 Firefox/18.0" "my-domain.com"
49.150.211.165 - - [10/Dec/2014:10:07:27 +0100] "GET / HTTP/1.1" 503 206 "-" "Mozilla/3.0 (Windows NT 5.1; rv:18.0) Gecko/20100101 Firefox/18.0" "my-domain.com"

[edited by: doc_z at 9:38 am (utc) on Dec 10, 2014]

[Leosghost]

My hosting company was also offering a professional solution but the price was very high (the price of a new car).

Change hosts..

[doc_z]

Changing hosting would takes days because there are several databases with gigabytes of data, cronjobs and so on. Moreover, you have to find a hoster (in Germany) which not only offering protecting against Ddos but also fulfill the other technical requirements.

I got a hint from my hosting company for a professional service offering protection against Ddos attacks. This was easier to realize because I just have to change some DNS entries. Now the request are filtered before reaching my server.

[aristotle]

I doubt that another hosting company would want a site that's under such a massive attack. As for your current hosting company, what are they doing about the other sites on your server? I guess I don't understand how hosting companies usually handle this kind of situation.

[doc_z]

As for your current hosting company, what are they doing about the other sites on your server?

Good question... of course, I cannot answer it.

[lucy24]

what are they doing about the other sites on your server?

The host must be doing something, because I note that all the examples in the log snippet are getting 503 responses. (Mine seizes up at something like 30 concurrent requests; you can see it in error logs. In years past I've had the occasional Bezeq robot get slightly throttled that way.)

If you haven't already looked at your error logs, do so. They generally live in the same directory as ordinary access logs, and should be available in the same way.

It should be noted that there's nothing you can do to stop unwanted visitors from making a request; you can only block it. A firewall-- if it were your own server-- can stop them just outside the server door, so at least they're not cluttering up the logs. Kinda like an unwanted human visitor. You don't have to open the door, but you can't stop them from knocking. Even if you get a restraining order (firewall) you can't prevent them from waving their placards 101 feet away from your front door.

But that's why I suggested trying alternative routes such as a 301-to-self or a manual 404. Sometimes this can work, so it's worth trying if it has been going on for a long time. It's very unusual for this kind of thing to go on for more than a few hours unless it's targeted and intentional.

The hosts themselves can't possibly want this to continue. The whole server must be getting ready to melt down, and other sites on the server are bound to be affected.

[keyplyr]

If the host is returning a 503, why all the fret? The host is doing what they want to do with it. As a client of this host, it is out of your hands. There's nothing you can do unless you want to move.

[wilderness]

If the host is returning a 503, why all the fret?

Ditto!

[not2easy]

Co-incidence that both of the IPs shown resolve to telecoms? I expected to see at least one server farm.
31.176.216.0 - 31.176.223.255
netname: BIHNET-SE800-ZE3
BH Telecom BRAS Bihac PPPoE dynamic
Bosnia and Herzegovina

49.144.0.0 - 49.151.255.255
netname: IPG
Philippine Long Distance Telephone Company

[wilderness]

FWIW

31.176.221.198 NT 6.6;

Sure thing!

[lucy24]

Philippine Long Distance Telephone Company

Funny, I've met a few Philippine robots just lately too. But I tend to assume that computers in some parts of the world are simply more infection-prone, as if there were a direct correlation between "Is it safe to drink the water?" and "Is it safe to download files?"

This position may not stand up to closer scrutiny :(

[doc_z]

If the host is returning a 503, why all the fret?

This were just 2 examples of 10 requests. I had more than 3 billion requests and (surprise) I cannot post all of them. Not all requests were getting a 503. 503 was possible due to limiting the number of requests by IP.

However, even 503 might cause problems when you're having millions per minute (log files for example) and limiting the number of request by IP was causes several other problems.

Btw, the attack ended after 5 days. Most IPs came from Turkey, Romania and Bulgaria.

[doc_z]

Just in addition: the attack continued on Saturday...

Anyhow, it's now under control.