MySql overload tracked down

[NielsSloth]

Hi

I made an earlier post concerning overload of MySql running on a Win2003 webserver, and by looking through the log files I found out what was happening.

It seems to be a series of IP adresses from Fremont CA, 66.220.20.* that was running through all the pages on one of my sites with ca. 30.000 photos of biological organisms (plants, butterflies etc.), at a rate of up to 10 requests/second, which combined with limited bandwith (756 kb/sec) was enough to make MySql lock up for all practical purposes.

I had to write a script to redirect the IP-adresses to a page where they do no harm; if anybody knows of a more general solution to the problem (perhaps a forced timelap for requests to the database), please let me know.

Does anybody recognize the IP-adresses, or have a clue to what has been happening? I guess it could have been some guys playing with development of a search robot, or maybe even a hack attempt. Is it legal to overload a site with automated requests like that? It certainly took me some time to find out why my server was performing so poorly.

Sincerely Niels from Denmark

[volatilegx]

> Is it legal to overload a site with automated requests like that?

That would depend on the location of the server and the place where the requests were coming from. Even if it is illegal, it's tough to get the police (or whatever the proper authority is) to prosecute this type of case unless massive worldwide damage is done.

[NielsSloth]

It is the same here in Denmark, and of course it is most probably not intentional.

One other possible explanation may perhaps be that the IP-adresses are a range of proxy servers supporting one user who is trying to download the entire website, so he can browse it locally. Some years ago there was a program called Webdog that could do that automatically.

Anyway, I am really happy that I found the reason and solved it, I had 10 sites down for a couple of days, and that is not very nice.

Niels from Denmark

[antirack]

[webmasterworld.com...]

[NielsSloth]

Thank you for the information about the earlier thread about the same problem, I can see that I am not alone.
If I get the overload problem again, I shall have to implement the scripts suggested in .asp, which is what I use.
I will probably make some kind of delay, so no IP can get anything more often than once/second, perhaps with a couple of 'free rides' in a row, that are not delayed, but I have to read the above thread carefully, since the people there did a good job of testing and implementing different solutions, as far as I can see.

Thanks

/Niels from Denmark

[stu_uk]

it sounds to me like it was a search bot trying to index your site.

i wouldnt have though 10 hits per second would be enough to lock up mysql, many forums use mysql as a backend and no doubt they recieve a LOT more than 10 reqs per second.

one thing i know can cause problems is if you were storing pictures/sounds/videos as BLOBs in the database. Also check to make sure that are not performing full table scans unecessarily.

[NielsSloth]

I am not storing photos in the database, only filenames.
For each pageload from the site a database query is run in a table with ca. 30.000 entries, each with 20-30 fields, and that kicks up the CPU momentarily at 50%, on a 3 Gigaherz P4 server with 4 GB of Ram.
If that is not normal, I am very interested in advice for better database performance.

Thanks

Niels from Denmark