|Cuil killing sites?|
Crawler going beserk
Apparently Cuil has not had its full share of negative publicity yet. An excerpt from the full article at techcunch [techcrunch.com]:
An anonymous tipster wrote to us this morning to tell us that Cuil, the ill-fated “Google Killer,” has unleashed its Twiceler indexing bot on websites across the globe and in the process, has brought many sites down.
“I don’t know what spawned it, but when Cuil attempts to index a site, it does so by completely hammering it with traffic,” the tipster wrote. “So much, that it completely brings the site down. We’re 24 hours into this “index” of the site, and I’ve had to restrict traffic to the site down to 2 packets per second, while discarding the rest, or otherwise it makes the site unusable.”
Has anybody here had any problems with Cuil / twiceler hammering their sites into oblivion? If so, what are you doing to counter it? Blocking Cuil?
I'm no fan of kwill, but it does annoy me when bloggers generalize from one site to the whole darn Internet.
If it were one tenth true, then why don't the idiots block the bot?
If it were one twentieth true, then maybe, er, another site or two might have reported being 'hammered'.
Chances are, this one anonymous site has a poor server setup, and should be thanking kewell for showing them they have issues.
Even Google gets featured here as 'killing sites by overspidering' - but once you think about it, it's got to be twaddle.
But I do believe the stories that kwoll is ignoring robots.txt and has a pretty cr*p spider ... but that's a matter of record from non-anonymous non-bloggers ;)
Yep- months ago, before they even launched. I think we contacted them and asked them to stop hitting our sites.