|Quality Control and Internal Policing at DMOZ|
How does the ODP implement and maintain QC for a massive 7,000+ editor project?
Simple enough. The title pretty much says it all.
Quality control. Quality assurance. Rooting out self-interest.
Anyone care to comment?
How has the process of assuring quality control changed or improved over time?
How proactive is the process?
Any recent - in the past 12-24 months - improvements in DMOZ's quality assurance program? Such as?
IF I was aware of a section of DMOZ that was suspect for self-interest and for "doing buddies favors" what would be the best method for helping with the housecleaning?
Every editor can see the logs that list everything that happened in a category, and everything that each individual editor has done. They are useful to find any problem patterns of editing, as are other tools.
There are a number of quality control processes that go on all the time. At one point, a group of editors went on a mission to find and fix as many spelling errors as possible. In the process they visited thousands of categories and things were "noticed" and looked into in more detail in a number of those.
There are plenty of internal mechanisms for reporting inappropriate listings, and potential problems with either categories, or editors.
Any editor can check up on any site, category, or editor any time they want to do so. There are plenty of ways to take things further if they see something that doesn't look right, and there are hundreds of editors that can fix those problems on the spot.
There are many ways to get information, and many ways to communicate, most of which will never be documented outside of the community.
There is a forum thread in the editor's public forum for people to report obviously duff listings (many of which are merely link rot, or bait and switch sites), and there is a Report Abuse form to report more serious things.
I would appreciate it if someone would post links to each offical DMOZ avenue or channel for reporting issues, filing a grievance, tracking action on a filed report, etc: Online forms, dedicated forum pages, etc.
I'd like to develop a "resource thread" that I can refer people to.
Any and all further details about DMOZ quality control and policing processes, policies, procedures is welcome.
There is only one place - google DMOZ abuse report
Apart from that, the rest is internal to DMOZ for a whole range of reasons (mostly so the abusers get very little information on how caught, so they can stay under the radar scope next time)
I am not longer a DMOZ editor and was never privy to abuse investigations, though I was able to check URL notes and was familiar with a number of abuse allegations. NONE of them turned out to be abuse (I could see the facts on who did what, to what URL, on what day).
I can never figure out why so many webmasters somehow think that DMOZ should be accountable to them! ... don't figure!
At the top of every category page (example [editors.dmoz.org]) there is a report abuse/spam link and an update listing link. The latter can be used by anybody, not just the website owner, to report QC issues.
Finally, there's a complete sub forum over at RZ for reporting QC matters.
What it actually does:
When people apply for a category - they aren't getting any major area to start with if they're accepted - a small category or subcategory to a subcategory ... and then the other editors above them can see what is done with that, how and why with the notes and work people make.. (sometimes there's a three page note on why something was done, other times a line - sometimes a five hundred pages thread in the forum - depends - if people are in doubt - they're usually told to go the forum and ask there to get an impartial opinion, and go with what other people tell them..you can also get a co-editor, and so it goes on ....
Those who has been in dmoz since it was that netscape directory back in the late 90's when the browser war between netscape and explorer was on its highest and the directory first began and nobody hardly knew about it - they have some experience with what works and what doesn't -
Furthermore the editors of any category are also obligated to state all their affiliations.. and they can be removed without notice from any and all categories they edit if they are found to have other affiliations than the ones they tell.. it's not illegal to add your friends site - or your own per se - you know - if its treated the same as any other page and listed based on that evaluation - and usually editors above you will tell you if you are out of line and delist the sites and so on and you will get a reprimande and a "no, don't do that .. or if its serious and not just a mistake/bad decision or three - they'll kick you out -
Best idea and what is commonly suggested if people are in doubt about something and how to go about things are to go to the forum and ask other people if they think this and that should be listed under this and that or if this or that should be edited or whatever - or ask the editor above them in the hierarchy if he/she could review this and that site since you are affiliated with them in one way or the other and cannot remain impartial on the review of it - but it's still relevant..
There's a way to tell if you think there is something going on which shouldn't be going on - by using the abuse submission - tell your story, and what you think is going on - if you have any kind of documentation - tell them to contact you through email and you will send it to them - and it'll be investigated by the meta's. Again - it's a longer proces, since anybody can accuse anybody of doing anything - but they do look at it - and also into it..
Besides from that - there's a huge discussion going on about this topic in the editors forum - with alot of ideas, suggestions and discussions about how it could be improved and get better - what it's going to end with - no idea :)
Gents, thanks for the insights and information. The explanations are in somewhat stark contrast to what one might assume if all one had to go on was the public snarling. (And we're not going there in this thread.)
What I'd like to know is a bit more about the editor vetting process.
How do you know when you've got a good one? How do you make that determination?
And what are the checks and balances in place to quickly determine if either a) Something was missed in the vetting process; or, b) if someone has been turned to the dark side?
Also, has DMOZ ever pulled down and rebuilt entire subsections due to "issues"?
I like the (potential) signs of life (AOL blog) and I'm wondering: What has been learned from the 1st ~10 years of editing and editors and how will that insight and knowledge be applied going forward?
Moderator's Note: I've initiated this thread to create a "resource thread" - so we ordinary folk may have a bit more insight into the internal workings and QC at DMOZ. This is not a thread to insert critiques or criticism. Y'all can start another thread to prove "it ain't so", if you wish, but that's not this thread. If anyone has a question in the same spirit - a bona fide, non-hidden agenda gripe pseudo-question - about maintaing QC in such a large scale project I invite you to post it up. Thank you all for your cooperation, including those who really really really would like to pounce . . but don't. Kudos to you too. :)
This thread is about "How DO YOU manage 100s or 1000s of active editors, 10s of 1000s of submissions a month or a year, and manage to do so well enough that Google thought/thinks y'all worthy of inclusion in their system"?
What does it take to maintain growth and quality control of the largest human edited directory?
[edited by: Webwork at 10:29 pm (utc) on Oct. 2, 2007]
One of the most important elements of QC is ... public review. All the listed sites are visible: the ones that shouldn't have been listed, the ones that were repurposed immediately upon being listed, the organizations (or businessmen) who died last week (or last year...)
It's hard to talk about what "quickly" means in such a context, but reports of bad listings are pretty consistently reviewed within hours or days. (Obviously the RESPONSE doesn't always come as quickly, but in clear-cut cases, the first response may be repairing the directory.)
This is worth emphasizing for several reasons.
(1) ANYONE can help. You don't have to show linguistic fluency, or establish trust. All you have to do is look, and report what you see.
(2) The response is (for all practical purposes) completely public--anyone can see what was done, and anyone can watch for patterns of repeat damage.
(3) These kind of problems are both the most common and the most serious problems the directory has--beside these, all other problems are insignificant in number of occurrences, and in severity per occurrence.
Some areas of the directory have been completely reorganised, several times. When a category is initially started, all relevant sites are placed there.
Once the category grows, it becomes obvious that it needs to be split into different sub-categories. Problem is that another sub-category, in another branch, might duplicate that topic again. In that case, discussions may go on for many weeks as to the best way to merge, split, and reorganise the whole lot.
When editors have left, or been pushed, then some have left a complete mess behind. Someone has to volunteer to go and clean it up - taking resources away from some other part of the directory while they do it.
You're not going to get much said on the editor application process, as there are too many people who try to circumvent it. Suffice to say, that the reviewer will usually know the right person when they see one. The application form is really your first three edits for the directory. They demonstrate if you have a clue as to what editors actually do.
Figures have never previously been discussed in public, but AOL staff have recently hinted in the ODP blog that about 10 new editors per day actually join the directory, worldwide. At least several of those will be outside the UK/US English language parts of the directory, of course.
There is a huge sense of community inside the ODP, and I would guess probably close to 1 300 000 posts in the internal forums (i.e. more than half the size of WebmasterWorld forums). Is that a community going nowhere? I think not.
When I'm evaluating editor applications, I can summarise what I'm looking for as follows. Please don't ask me for details.
Communications skills - the ability to understand and express oneself fluently in the language of the category.
Understanding of the category - demonstrated by the ability to find and describe appropriate new websites for it.
Attention to detail
If you replace category with job in the above, it's pretty similar to the list you'd use when evaluating resumes in a real life employment situation.