Lord_Majestic - 5:08 pm on Jan 7, 2008 (gmt 0)

If they are using nutch, then they won't get far as many block this UA. Better come up with their own UA.

I think this is the least of their worries as Nutch allows change of UA.

The issue really is not to crawl billions of pages, but to actually make sense of them to show Top 10 very good results from such a big index. This is the hard bit and Nutch itself does not solve it - one would have assumed that this is exactly where Wikia can help by throwing money they raised from investors into exactly this kind of work, but so far it seems that Wikia just took Nutch and added some GUI on top of it.

New fancy GUI does not win the market - A9 (from Amazon who incidentally listed as investor in Wikia) used Google's code and database (under license), yet even though they did not have problem with the relevancy, they had problem of differentiation - if people get no better result than on Google then they won't switch, no matter how fancy GUI is.