Cuil

[ning]

Anyone know the Agent for the new search engine Cuil?

[cuil.com...]

[incrediBILL]

It's "twiceler" and they have more information about it here:
[cuil.com...]

You can also find lots of posts with details about it by searching WebmasterWorld [google.com].

[incrediBILL]

Just in case anyone missed it, Cuil launched [webmasterworld.com] to mostly unfavorable reviews.

[ning]

Thanks Bill, that's why it's worrying me. It sucks bandwitdh and it spiders the most crazy things.

I am running a big forum, and it only spiders irrelevant things like "post" "print" and things like no other spiders do (blocked by roboats.txt and the software itself).

Will block them now I think.
Cheers

[GaryK]

---

System: The following message was spliced on to this thread from: http://www.webmasterworld.com/search_engine_spiders/3744847.htm [webmasterworld.com] by incredibill - 1:37 pm on Sep. 16, 2008 (PST -8)

---

Twiceler/Cuil used to be hosted at Hurricane Electric. It's not anymore. Do you all still consider it a bot worthy of being banned? At least this time when it visited it read robots.txt.

BTW, I laughed myself silly at this claim on their home page:

Cuil is the biggest search engine on the planet.

[jdMorgan]

They simply claim to have indexed more URLs than anyone else, so the claim is as valid as any advertisement you might read, see, or hear...

Only time will tell, but at least they've now published their index, which makes it easier to justify allowing them to crawl our sites.

Jim

[GaryK]

Jim, it's one thing to claim to have indexed more URLs than anyone else and being the biggest search engine on the planet. :)

blocked by roboats.txt

Is this a special file for maritime industry websites? LOL ;)

~gary.

[Samizdata]

Is this a special file for maritime industry websites?

It might be Cuill's cunning response to Google's patented floating datacenters.

...

[leadegroot]

---

System: The following message was spliced on to this thread from: http://www.webmasterworld.com/search_engine_spiders/3765969.htm [webmasterworld.com] by incredibill - 12:53 pm on Oct. 15, 2008 (PST -8)

---

Had to ban the cuil agent Twiceler last night when it was happily trawling through directories that everyone else seems to manage to stay out of, as instructed by the robots.txt

Anyone else noticed this?

Bad Cuil, bad! :(