Forum Moderators: open
72.249.60.74 - - [19/Jul/2008:10:20:13 -0400] "GET /robots.txt HTTP/1.0" 200 2468 "-" "VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)"
They've apparently come a long way in the two years since the thread above was posted. I can confirm that they *do* obey robots.txt in the case of
User-Agent: *
Disallow: /
One of the more interesting things about this engine-to-be is that they are currently exposing their basic search algorithm to view. They're apparently using keyword inverse document frequency, raw keyword frequency, and keywords-in title, and combining those along with the same calculations for the entire keyphrase (likely excluding stop words) -- among other things. Keywords and phrases are apparently weighted according to their frequency of occurrence on the Web.
The "ranking report" for each result URL is available by clicking the "Explain" link below each returned search result. An interesting peek behind the curtain; I'll bet it goes away when the "Beta" testing ends...
Jim
they are currently exposing their basic search algorithm to view
It is Nutch based.
