Welcome to WebmasterWorld Guest from 54.162.168.187

Forum Moderators: bakedjake

Message Too Old, No Replies

searchme.com ignoring my robots.txt

Problems with Charlotte bot ignoring robots.txt contents

     
11:03 am on Feb 24, 2009 (gmt 0)

New User

10+ Year Member

joined:Feb 8, 2005
posts: 18
votes: 0


Hello,

I've had Problems with the Charlotte-Bot (from searchme.com) - they have taken down my web server repeatedly (admitted, it's a slow machine, but no other search engine has yet managed to bring the machine down).

I've contacted them - no response (after 6 days).

I've added the following to my robots.txt (according to the info on their page):

User-Agent: Charlotte
Disallow: /
Crawl-Delay: 30

They fetched robots.txt about every hour, but they ignore their instructions. Therefore, my only choice was to block their network ranges on the IP level (which i have done now) - And i don't really bother about them now, because i have never received a *single* referral from them.

Plus, i do consider their screenshotting technique content theft anyways (contrary to other "visually enhanced" search engines, you can read the full text on their screenshots, so there's no need to visit the original page anymore..

Anybody else experiencing issues with them? My feeling is they are slightly "challenged" in running a crawler... any thoughts appreciated.

12:05 pm on Feb 24, 2009 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:7518
votes: 506


Gave them a try. Liked it when one of the fellows came on the list and chatted... but after so many months of letting them beat me up bandwidth-wise and NOT ONE REFERRAL just gave 'em the boot via robots.txt. Will wait and see if that is honored.
2:57 pm on Mar 2, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 16, 2003
posts:992
votes: 0


I've also banned them via robots.txt on some sites, and I'm waiting to see if the situation changes in any way. My guess is they will not take off, because of the screenshot infringement. It's not as though sensible webmasters will be falling over each other to recommend them to other people.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members