homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

    
searchme.com ignoring my robots.txt
Problems with Charlotte bot ignoring robots.txt contents
nonanet




msg:3856503
 11:03 am on Feb 24, 2009 (gmt 0)

Hello,

I've had Problems with the Charlotte-Bot (from searchme.com) - they have taken down my web server repeatedly (admitted, it's a slow machine, but no other search engine has yet managed to bring the machine down).

I've contacted them - no response (after 6 days).

I've added the following to my robots.txt (according to the info on their page):

User-Agent: Charlotte
Disallow: /
Crawl-Delay: 30

They fetched robots.txt about every hour, but they ignore their instructions. Therefore, my only choice was to block their network ranges on the IP level (which i have done now) - And i don't really bother about them now, because i have never received a *single* referral from them.

Plus, i do consider their screenshotting technique content theft anyways (contrary to other "visually enhanced" search engines, you can read the full text on their screenshots, so there's no need to visit the original page anymore..

Anybody else experiencing issues with them? My feeling is they are slightly "challenged" in running a crawler... any thoughts appreciated.

 

tangor




msg:3856534
 12:05 pm on Feb 24, 2009 (gmt 0)

Gave them a try. Liked it when one of the fellows came on the list and chatted... but after so many months of letting them beat me up bandwidth-wise and NOT ONE REFERRAL just gave 'em the boot via robots.txt. Will wait and see if that is honored.

Rosalind




msg:3860966
 2:57 pm on Mar 2, 2009 (gmt 0)

I've also banned them via robots.txt on some sites, and I'm waiting to see if the situation changes in any way. My guess is they will not take off, because of the screenshot infringement. It's not as though sensible webmasters will be falling over each other to recommend them to other people.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved