homepage Welcome to WebmasterWorld Guest from 54.227.215.140
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Cuil
ning

10+ Year Member



 
Msg#: 3709085 posted 11:26 am on Jul 28, 2008 (gmt 0)

Anyone know the Agent for the new search engine Cuil?

[cuil.com...]

 

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3709085 posted 6:18 pm on Jul 28, 2008 (gmt 0)

It's "twiceler" and they have more information about it here:
[cuil.com...]

You can also find lots of posts with details about it by searching WebmasterWorld [google.com].

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3709085 posted 6:56 pm on Jul 29, 2008 (gmt 0)

Just in case anyone missed it, Cuil launched [webmasterworld.com] to mostly unfavorable reviews.

ning

10+ Year Member



 
Msg#: 3709085 posted 7:11 pm on Jul 29, 2008 (gmt 0)

Thanks Bill, that's why it's worrying me. It sucks bandwitdh and it spiders the most crazy things.

I am running a big forum, and it only spiders irrelevant things like "post" "print" and things like no other spiders do (blocked by roboats.txt and the software itself).

Will block them now I think.
Cheers

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3709085 posted 5:08 am on Sep 15, 2008 (gmt 0)


System: The following message was spliced on to this thread from: http://www.webmasterworld.com/search_engine_spiders/3744847.htm [webmasterworld.com] by incredibill - 1:37 pm on Sep. 16, 2008 (PST -8)


Twiceler/Cuil used to be hosted at Hurricane Electric. It's not anymore. Do you all still consider it a bot worthy of being banned? At least this time when it visited it read robots.txt.

BTW, I laughed myself silly at this claim on their home page:

Cuil is the biggest search engine on the planet.

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3709085 posted 10:46 pm on Sep 16, 2008 (gmt 0)

They simply claim to have indexed more URLs than anyone else, so the claim is as valid as any advertisement you might read, see, or hear...

Only time will tell, but at least they've now published their index, which makes it easier to justify allowing them to crawl our sites.

Jim

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3709085 posted 11:36 pm on Sep 16, 2008 (gmt 0)

Jim, it's one thing to claim to have indexed more URLs than anyone else and being the biggest search engine on the planet. :)

blocked by roboats.txt

Is this a special file for maritime industry websites? LOL ;)

~gary.

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3709085 posted 11:55 pm on Sep 16, 2008 (gmt 0)

Is this a special file for maritime industry websites?

It might be Cuill's cunning response to Google's patented floating datacenters.

...

leadegroot

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3709085 posted 9:59 am on Oct 15, 2008 (gmt 0)


System: The following message was spliced on to this thread from: http://www.webmasterworld.com/search_engine_spiders/3765969.htm [webmasterworld.com] by incredibill - 12:53 pm on Oct. 15, 2008 (PST -8)


Had to ban the cuil agent Twiceler last night when it was happily trawling through directories that everyone else seems to manage to stay out of, as instructed by the robots.txt

Anyone else noticed this?

Bad Cuil, bad! :(

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved