ShunixBot

New French search engine

[GaryK]

User Agent:
Mozilla/5.0 (compatible; ShunixBot/1.1; [shunix.com...]

IP:
80.119.212.*

The site is still under construction. It reads robots.txt but it requests pages that have not been on my server for over a year.

[GaryK]

This bot showed up with a new user agent this week:

XunBot/Xun 1.8.9 (+http://www.shunix.com/bot.htm)
82.232.240.192

It read a huge portion of one of my sites, including disallowed files, before it even read robots.txt. Once it did read robots.txt it seemed to respect it. They seem to have fixed their problem with requesting non-existent files.

[DanA]

This bot is at a pre-alpha stage.
It is tested according to Google searches. The programmers' difficulties with perl are exposed in the French developers perl forum at www.developpez.net
It seems to follow robots.txt once found and doesn't limit bandwidth usage.
Also uses MonRobot/1.1 in its UA.

[GaryK]

Thanks for the info Dan. :)

The whole issue with their programmers is sad and unprofessional. Until they learn how to write code I wish they'd do their pre-alpha testing on their own servers.

This is the sort of issue that can doom a start-up before it really gets going. Get a bad reputation amongst webmasters, have your bot banned, and wind up with inferior search results that will keep customers away in droves.