206.16.1.x (cnet) killing us

[papamaku]

Got hit big time by cnet/search.com coming in on ips 206.16.1.x. It hammered us so hard it was like a full on DOS attack, nothing else could get on.

Have blocked them now, which is a shame as we like getting crawled, but not if its gonna stop users getting on the site.

Also makes me wonder what cnet are up to? They seemed happy to just be a meta - searching google, ask etc, but maybe now they're thinking of getting into the index battle.

maku

[amznVibe]

Did they obey robots.txt? Use a referer or user agent? Cnet owns search.com which has 206.16.1.111 - 206.16.1.224 reserved. But is it a meta engine, until now?

[MarieC]

What was the UA? If you don't mind saying.

[papamaku]

only thing that came up in the logs was:

Mozilla/4.0+(SavvySearch)

not much of a UA + i couldn't find any requests by them for a robots.txt (but we dont have one anyway)

[amznVibe]

That's enough of a UA to know that something is up (along with the IP, it seems legit).

Search.com *is* savvysearch (they bought them for $22 million in October 1999).

But I can't find any mention of them doing their own spidering. So this is new perhaps.