Forum Moderators: open
However, I do not think it would take very long before people find ways to confuse or defeat the AI so that their pages are given precedence over others'. Second, the more specialized the taxonomy, the more complex the matching will need to be, especially for subjects that are known by multiple names. But I wish you a lot of luck.
wow just checked that out and we've played around with the Grokker software. I'm sure adding graphics to our potential matches might make this project 10x more visually appealing (wont really effect the bottom line, but might hit that "cool" effect). Anyone any good at creating that type of visual enviroment? Plan on potentially having something that the public can toy around with by late late tonight/early tommorow.
If you don't use DMOZ, you might want to look at wordnet. Totally open source so if its useful to your app, it wont hinder any future commercial aspect of it.
btw would love the URL if you ever get it online :)
Thank you for the advice!
**Added
We are looking for an open source of the categories themselves, we already have the technology behind understanding the words and placement, we just need to know where to file them without legal worries. DMOZ has a perfect cateogorization strucutre for this endeavor.
Going to work now. Hopefully the team has somthing to show later this evening.
... the ability to categorize all 4 billion pages in Google ...
I wonder if you are going to categorize the pages or the sites.
Keep us updated Giga.
don't just hype it up and name build, let us test it :)
Can you add this site in your profile, or sticky me the url?
So Many Questions to be answered to categorize a site properly(that is why it should be left to the webmaster of the site).
What is the site about,
widget manufacturing
widget wholesale
widget retail
replica widgets
Midget widgets
widget repair locally with widget parts sales
widget parts sales worldwide
used widgets
two wheeled widgets
three wheeled widgets
chopper widgets
custom widgets
porn widgets or safe sex widgets
site blogging about widgets
site slamming widgets because it does not like them
site with the complete history of widgets, the different models, how to chop widgets, how to get dates on widgets, tips and tricks on racing widgets, with diagrams how to do it.
Do you just have one big category for widgets, and throw every site in it. Do you break it down in regions for local only?
Do you break it down between commercial and informational?
What do you do with the sites that do not even have the keywords in the front page (top in G in a lot of cases)?
Language Translation?
Will it be a one time shot, or will it return to pick up info as the site changes?
The editors of the ODP (just an example, could be any human edited directory) must continually face these questions, and re-categorize, and judge where a site should be placed within the directory. Even human editors struggle with these questions to properly place a site, and even continually move sites that other editors have already categorized.
Questions, Questions Questions, Equate To Processing, Processing, Processing. Better Get Lots of boxes!
Sticky me the url, I wanna see it work.
Thanks!
Thumpcyc
I am a developer working on this auto categorizer project.
To answer your questions, thumpcyc, yes it is intellignt enough to intelligently differentiate "widget sales", "widget manufacturing", "guide to widgets", and so on.
Right now I am working on integrating categories based on DMOZ's setup into the demo, rather than the current structure more suited to a yellow pages type of directory. With more categories available for this system to use, the accuracy will increase even more.
But rather than just talking about it, we want everyone here to actually see this in action. I anticipate having the final demo with DMOZ-based categories running later tonight, at which time we will sticky everyone with a URL so you can see it for yourselves.
*update, import finalized, working on the first category... ADULT..hmmm not my first pick of test data...but interesting work for us enough :P
We have responded via sticky to everyone who has requested access.
I didn't get your sticky.
Hope I did not ask so many questions that I scared you.(I really held back, I had many more, I was waiting to see if the site would answer some of them).
I also did not get a sticky mail with the url.
When it is ready, I wanna see it. (even if it is not ready, I'll check it out)
(you should also give an overview of what you want people to test, what to expect, and a way to provide feedback to enhance the end results)
Thanks!
Thumpcyc
[edited by: skibum at 9:02 am (utc) on Mar. 13, 2004]
Also, you are welcome to elaborate more about the technology behind all this :-)
Cannot post the results here (I sent them to Giga e-mail), Cannot post specific urls and cats.
Thanks for letting me check it out, let me know when the algo has changed sufficiently for a retest.
Later,
Thumpcyc
it correctly classified 2 sites out of the dozen I submitted. It totally failed to classify two of them(easy ones)and incorrectly classified the rest.
Hmmm, 1 out of 6?
Maybe the algo plays dice. ;-)
The premise of this project is to disagree with that exact statement :). We intend to categorize sites BETTER than DMOZ is able to, due to the content of the pages. We have found mixed results so far, but the trend is looking good for this outcome. I know that sounds crazy, "how can a script better classify a site than a human could?". Well here is how and why. DMOZ has over 450,000 different categories. NO HUMAN is able to look or remember all 450,000 potential matches for a website based on its content. Although a human editor might say, hmm this site would best fit this category, he may be unaware of another category outside his / her field that is a better match under a different hierarchy. Yes nothing can compete with a human's mind, however even a dumb machine can do a better job than some of these experts whose knowledge doesn’t go beyond the few categories they represent. I'm going to look for a few examples of this premise. I don’t mean to offend ANYONE who is a DMOZ editor, just a unique way of looking at the directory from our perspective. Humans are very good with very specific topics and similar data batches in general (typically don’t hear about someone getting a bachelors degree in EVERY field, usually just 1), however machines handle countless topics and data much more efficiently.
No we are using the resulting data, and end users search to auto train the categorizer. Want to stress this is a pure database, and other than the catgories, we have no connection to the websites or their placement in DMOZ (that would be cheating) :)
