Welcome to WebmasterWorld Guest from 54.205.209.95

Forum Moderators: bakedjake

Message Too Old, No Replies

Cuil killing sites?

Crawler going beserk

   
11:28 am on Sep 3, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Apparently Cuil has not had its full share of negative publicity yet. An excerpt from the full article at techcunch [techcrunch.com]:


An anonymous tipster wrote to us this morning to tell us that Cuil, the ill-fated “Google Killer,” has unleashed its Twiceler indexing bot on websites across the globe and in the process, has brought many sites down.

“I don’t know what spawned it, but when Cuil attempts to index a site, it does so by completely hammering it with traffic,” the tipster wrote. “So much, that it completely brings the site down. We’re 24 hours into this “index” of the site, and I’ve had to restrict traffic to the site down to 2 packets per second, while discarding the rest, or otherwise it makes the site unusable.”

Has anybody here had any problems with Cuil / twiceler hammering their sites into oblivion? If so, what are you doing to counter it? Blocking Cuil?

11:12 am on Sep 4, 2008 (gmt 0)

WebmasterWorld Senior Member quadrille is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I'm no fan of kwill, but it does annoy me when bloggers generalize from one site to the whole darn Internet.

If it were one tenth true, then why don't the idiots block the bot?

If it were one twentieth true, then maybe, er, another site or two might have reported being 'hammered'.

Chances are, this one anonymous site has a poor server setup, and should be thanking kewell for showing them they have issues.

Even Google gets featured here as 'killing sites by overspidering' - but once you think about it, it's got to be twaddle.

But I do believe the stories that kwoll is ignoring robots.txt and has a pretty cr*p spider ... but that's a matter of record from non-anonymous non-bloggers ;)

3:19 pm on Sep 4, 2008 (gmt 0)

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Yep- months ago, before they even launched. I think we contacted them and asked them to stop hitting our sites.