homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

searchme.com ignoring my robots.txt
Problems with Charlotte bot ignoring robots.txt contents

10+ Year Member

Msg#: 3856501 posted 11:03 am on Feb 24, 2009 (gmt 0)


I've had Problems with the Charlotte-Bot (from searchme.com) - they have taken down my web server repeatedly (admitted, it's a slow machine, but no other search engine has yet managed to bring the machine down).

I've contacted them - no response (after 6 days).

I've added the following to my robots.txt (according to the info on their page):

User-Agent: Charlotte
Disallow: /
Crawl-Delay: 30

They fetched robots.txt about every hour, but they ignore their instructions. Therefore, my only choice was to block their network ranges on the IP level (which i have done now) - And i don't really bother about them now, because i have never received a *single* referral from them.

Plus, i do consider their screenshotting technique content theft anyways (contrary to other "visually enhanced" search engines, you can read the full text on their screenshots, so there's no need to visit the original page anymore..

Anybody else experiencing issues with them? My feeling is they are slightly "challenged" in running a crawler... any thoughts appreciated.



WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

Msg#: 3856501 posted 12:05 pm on Feb 24, 2009 (gmt 0)

Gave them a try. Liked it when one of the fellows came on the list and chatted... but after so many months of letting them beat me up bandwidth-wise and NOT ONE REFERRAL just gave 'em the boot via robots.txt. Will wait and see if that is honored.


WebmasterWorld Senior Member 10+ Year Member

Msg#: 3856501 posted 2:57 pm on Mar 2, 2009 (gmt 0)

I've also banned them via robots.txt on some sites, and I'm waiting to see if the situation changes in any way. My guess is they will not take off, because of the screenshot infringement. It's not as though sensible webmasters will be falling over each other to recommend them to other people.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved