Welcome to WebmasterWorld Guest from

Forum Moderators: bakedjake

Message Too Old, No Replies

Cuil killing sites?

Crawler going beserk

11:28 am on Sep 3, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 17, 2004
votes: 0

Apparently Cuil has not had its full share of negative publicity yet. An excerpt from the full article at techcunch [techcrunch.com]:

An anonymous tipster wrote to us this morning to tell us that Cuil, the ill-fated “Google Killer,” has unleashed its Twiceler indexing bot on websites across the globe and in the process, has brought many sites down.

“I don’t know what spawned it, but when Cuil attempts to index a site, it does so by completely hammering it with traffic,” the tipster wrote. “So much, that it completely brings the site down. We’re 24 hours into this “index” of the site, and I’ve had to restrict traffic to the site down to 2 packets per second, while discarding the rest, or otherwise it makes the site unusable.”

Has anybody here had any problems with Cuil / twiceler hammering their sites into oblivion? If so, what are you doing to counter it? Blocking Cuil?

11:12 am on Sept 4, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member quadrille is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Feb 22, 2002
votes: 0

I'm no fan of kwill, but it does annoy me when bloggers generalize from one site to the whole darn Internet.

If it were one tenth true, then why don't the idiots block the bot?

If it were one twentieth true, then maybe, er, another site or two might have reported being 'hammered'.

Chances are, this one anonymous site has a poor server setup, and should be thanking kewell for showing them they have issues.

Even Google gets featured here as 'killing sites by overspidering' - but once you think about it, it's got to be twaddle.

But I do believe the stories that kwoll is ignoring robots.txt and has a pretty cr*p spider ... but that's a matter of record from non-anonymous non-bloggers ;)

3:19 pm on Sept 4, 2008 (gmt 0)

Moderator from US 

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 10, 2005
votes: 76

Yep- months ago, before they even launched. I think we contacted them and asked them to stop hitting our sites.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members