Forum Moderators: open
robots.txt? NO
Referer spammer? YES: [rexyo.com...]
The Netherlands-based 'fastest, newest' SE in beta "invitation" stage. Launching...soon.
On the 14th it hit a few of my sites as 2.03. Last night it came back as this:
Mozilla/5.0 (compatible; Rexyobot/2.04; http://www.example.com)
213.148.250.?
d594fa48.ftth.concepts.nl
No robots.txt text, just the default root page for each site.
Every request included referrer spam.
85.113.243.125
RexyoBot 1.11
robots.txt? NO
(Also through concepts.nl)
RexyoBot 1.11
First seen: 11/8/2008
Last seen: 11/16/2008
Total Visits: 5
ADDED: Oh, I also have:
Mozilla/5.0 (compatible; Rexyobot/1.09; [rexyo.com...]
Mozilla/5.0 (compatible; Rexyobot/1.12; [rexyo.com...]
[edited by: GaryK at 6:36 pm (utc) on June 21, 2009]
A while back (in 2007) the RexyoBot was released at version 1.x Last Month (May 2009) we upgraded our webcrawler and released version 2.x to satisfy our users with a free invitation. We plan to invite a couple of hundred individuals.
Setting the referer link is not an uncommon technique for searchengines across the web. The bigger searchengines from companies like Google, Yahoo and Microsoft all make use of this referer technique.
Respectfully, the way you're doing the link now is seen by many of us as link spam. The way you used to do it, with a link to your bot page, was far more acceptable.
[edited by: GaryK at 7:15 pm (utc) on June 29, 2009]
2.) In my experience, "setting the referrer link" to a bot's base URL (a.k.a. log-spamming) is all too common among smaller (as opposed to bigger) search engines.
With the exception of MSN's fake referers [webmasterworld.com], Google, Yahoo and Microsoft include an individual's search phrases in their referers and thus the entries are not bot-PR log spam but useful info.
Seeing as how your base URL is already in your UA, why use it to referer-spam, too?
Checking for the file "robots.txt" has always been considered important by the owners and will always be on the list of features. In fact, RexyoBot 1.09 through 1.13 all had this feature.
For now, RexyoBot is configured to only crawl homepage indexes. It will not go deeper than your homepage url: www.example.com, so the robots.txt feature is turned off.
However, when RexyoBot will start crawling deeper into websites, the robots.txt feature will be finished by our engineers and of course turned on.
Rexyo would really like to get some beta-testers to join us as soon as it goes live. Setting the referer link provides us a way to invite our future users. As we generate more traffic, the referer feature may be configured differently.
[edited by: Pfui at 10:05 pm (utc) on June 29, 2009]
It also includes some interesting new features. The referer feature has been turned off as the number of available Rexyo Beta invitations has declined.
Both thanks for your posts on RexyoBot, Pfui and GaryK
