homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

Cuil killing sites?
Crawler going beserk

WebmasterWorld Senior Member 10+ Year Member

Msg#: 3737220 posted 11:28 am on Sep 3, 2008 (gmt 0)

Apparently Cuil has not had its full share of negative publicity yet. An excerpt from the full article at techcunch [techcrunch.com]:

An anonymous tipster wrote to us this morning to tell us that Cuil, the ill-fated “Google Killer,” has unleashed its Twiceler indexing bot on websites across the globe and in the process, has brought many sites down.

“I don’t know what spawned it, but when Cuil attempts to index a site, it does so by completely hammering it with traffic,” the tipster wrote. “So much, that it completely brings the site down. We’re 24 hours into this “index” of the site, and I’ve had to restrict traffic to the site down to 2 packets per second, while discarding the rest, or otherwise it makes the site unusable.”

Has anybody here had any problems with Cuil / twiceler hammering their sites into oblivion? If so, what are you doing to counter it? Blocking Cuil?



WebmasterWorld Senior Member quadrille us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 3737220 posted 11:12 am on Sep 4, 2008 (gmt 0)

I'm no fan of kwill, but it does annoy me when bloggers generalize from one site to the whole darn Internet.

If it were one tenth true, then why don't the idiots block the bot?

If it were one twentieth true, then maybe, er, another site or two might have reported being 'hammered'.

Chances are, this one anonymous site has a poor server setup, and should be thanking kewell for showing them they have issues.

Even Google gets featured here as 'killing sites by overspidering' - but once you think about it, it's got to be twaddle.

But I do believe the stories that kwoll is ignoring robots.txt and has a pretty cr*p spider ... but that's a matter of record from non-anonymous non-bloggers ;)


WebmasterWorld Administrator lifeinasia us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

Msg#: 3737220 posted 3:19 pm on Sep 4, 2008 (gmt 0)

Yep- months ago, before they even launched. I think we contacted them and asked them to stop hitting our sites.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved