Forum Moderators: bakedjake
Currently we're just a handful of developers working part-time. At this point Nutch is coded entirely in Java, however persistent data is written in language-independent formats so that, if needed, modules may be re-written in other languages (e.g., C++) as the project progresses.Nutch has not yet been tuned for quality. There are ten or twenty knobs that we can twiddle to adjust the ranking formula. We are developing software to do this tuning automatically, but the current code just contains guesses. With a little tuning we should be able to get results that are competitive with those of major search engines.
As of June, 2003, we have successfully built a 100 million page demo system. Unfortunately, we do not yet have enough hardware to support a public demo. Hopefully we will be able to add that in the next few months. Stay tuned.
WBF
How many web applications have gone under because Java is just too slow
In this day & age, since there are search engines out there that actually work unlike those of 'yesteryear' that were spam ridden, cloaked & auto generated pages in every commercial category (I put many of those there myself ;) ) this Nutch deal doesn't have a chance.
1) Java - too slow, Google builds their stuff in ML (machine language).
2) Propriatary database architecture -> to scale to "infinity" you have got to have a custom database application. Closest thing to that is Oracle - and nope, they won't spring for that.
3) Part time developers
Doesn't matter how many monkeys you throw at a project - look at Grub.org - distrubuted, bla, bla, bla. Bought by LookSmart, and still they can't even figure out how to deal properly with Robots.txt.
Sorry, but the track record here is *unless* the people doing it have some serious background in the development issues AND a familiarity with the community & what they want to see in an engine thing won't go anywhere, imho.
I could be wrong, of course - more competition is always a good thing.
Then again who are they trying to compete with?
What makes them better than everyone else?
I mean i get annoyed at adverts as much as anyone but in the end we are looking for good results?
There are not that many web sites that google,inktomi and the rest dont have. Basically they have to have an edge for me to even try them.
Im sorry to be so cynical, i hope they prove me wrong and be very successful
Jeremy, Grub didn't catch up for various reasons. looksmart is a commercial company and no one is dumb enough to beleive that using Grub will actually benefit the public but rather make Looksmart rich.
Also don't doubt the monkeys :)
Look what those monkeys made linux today and you'll see that lots of part time developers cando. They can get better results then any company and any budget in the world and this is a proven fact.
I personally trust nutch to be a leading SE. I'm sick and tired of google and all the rest. I miss the days of google before they were all commercial. Today, you log to google and you just can't avoid those adwords boxes. They are all over the frickin' page. All people want is a good UNBIASED SE and open source proved to be able to deliver.. so we'll see...
And there is one major hitch any open source search engine must address: How can they avoid that spammers are reading the code and are tuning their websites and linkfarms according to the ranking formula?
According to nutch.org, Tim O'Reilly is a director of The Nutch Organization. So maybe O'Reilly (the company) is one day using that software to run their own search engine. That would be a reasonable step for them. But then that O'Reilly engine wouldn't be different from any other engine that uses software that wasn't developed by themselves.
Google has made a great move not to show banner ads. Why are people against them for?.I dont care that google are making money out of this, why not? What is so wrong with it? Im sure quite a few are happy with adwords and adsense. For years they gave us good results and they continue to try to.
Too much tall poppy syndrome here, if you want something , go work hard and get it, i believe in open source, however there is no such thing as unbiased. Im sick of the big sites being number 1 too, but in the real world it is exactly the same, some win , some lose
most importantly, not everyone can win and be in the top 10 results
hmmm.. if the algo is open source then we are in for a bit of spam methinks.
Pages in such format are often usefull by the sheer merit of their plainness and structure.
I do believe you can provide unspammed open source ranking. You just have to rank in such a way that the "perfect spam" is a great content site.
Google is trying that, but they fail due to their proprietary algorithm. They guess, they adjust, but they keep missing.
With an Open Source algo, you'd have the "perfect spam" right up front and could ensure that the "perfect spam" (i.e. high ranking results) are also the perfect results you want to see at the top.
Let's not forget what "spam" in SERPs really is... it's the documents that the engine thinks are the absolute best for the searcher. So you cannot remove "spam" at all, you can only ensure that the best spem is also the best content and pages.
SN
There are certainly a number of challenges to be faced but I wouldn't say their choice of technology was a problem.
The current Nutch codebase does not use peer-to-peer methods. This is an interesting approach, but it is difficult to make large-scale P2P searches return results in less than a second, as people have come to expect of web search engines. However we encourage someone to start a P2P-based search effort as a Nutch sub-project." (Here [nutch.org]).
They seemingly plan on returning results in about a second...doesn't sound too slow. :)
Jordan
Remember MS earned its monopoly status because the market demanded a standard operating system, Google is in some ways acheiving a monopoly by being that much better than the rest. It would be all redundant if yahoo search and msn search met the standard set by Google; they could start by removing the sponsored results.
Sid
