homepage Welcome to WebmasterWorld Guest from 54.226.80.196
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Cuil
ning




msg:3709087
 11:26 am on Jul 28, 2008 (gmt 0)

Anyone know the Agent for the new search engine Cuil?

[cuil.com...]

 

incrediBILL




msg:3709498
 6:18 pm on Jul 28, 2008 (gmt 0)

It's "twiceler" and they have more information about it here:
[cuil.com...]

You can also find lots of posts with details about it by searching WebmasterWorld [google.com].

incrediBILL




msg:3710547
 6:56 pm on Jul 29, 2008 (gmt 0)

Just in case anyone missed it, Cuil launched [webmasterworld.com] to mostly unfavorable reviews.

ning




msg:3710553
 7:11 pm on Jul 29, 2008 (gmt 0)

Thanks Bill, that's why it's worrying me. It sucks bandwitdh and it spiders the most crazy things.

I am running a big forum, and it only spiders irrelevant things like "post" "print" and things like no other spiders do (blocked by roboats.txt and the software itself).

Will block them now I think.
Cheers

GaryK




msg:3744849
 5:08 am on Sep 15, 2008 (gmt 0)


System: The following message was spliced on to this thread from: http://www.webmasterworld.com/search_engine_spiders/3744847.htm [webmasterworld.com] by incredibill - 1:37 pm on Sep. 16, 2008 (PST -8)


Twiceler/Cuil used to be hosted at Hurricane Electric. It's not anymore. Do you all still consider it a bot worthy of being banned? At least this time when it visited it read robots.txt.

BTW, I laughed myself silly at this claim on their home page:

Cuil is the biggest search engine on the planet.

jdMorgan




msg:3745963
 10:46 pm on Sep 16, 2008 (gmt 0)

They simply claim to have indexed more URLs than anyone else, so the claim is as valid as any advertisement you might read, see, or hear...

Only time will tell, but at least they've now published their index, which makes it easier to justify allowing them to crawl our sites.

Jim

GaryK




msg:3745973
 11:36 pm on Sep 16, 2008 (gmt 0)

Jim, it's one thing to claim to have indexed more URLs than anyone else and being the biggest search engine on the planet. :)

blocked by roboats.txt

Is this a special file for maritime industry websites? LOL ;)

~gary.

Samizdata




msg:3745982
 11:55 pm on Sep 16, 2008 (gmt 0)

Is this a special file for maritime industry websites?

It might be Cuill's cunning response to Google's patented floating datacenters.

...

leadegroot




msg:3765971
 9:59 am on Oct 15, 2008 (gmt 0)


System: The following message was spliced on to this thread from: http://www.webmasterworld.com/search_engine_spiders/3765969.htm [webmasterworld.com] by incredibill - 12:53 pm on Oct. 15, 2008 (PST -8)


Had to ban the cuil agent Twiceler last night when it was happily trawling through directories that everyone else seems to manage to stay out of, as instructed by the robots.txt

Anyone else noticed this?

Bad Cuil, bad! :(

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved