Open Directory Editors

A couple quick questions...

[furl]

I have a couple questions for editors of the "Open Directory Project" (www.dmoz.org). I’m working on software that tests employee compliance with unstructured rule bases (jargon, not a dig). The Open Directory Project would be a cool test case. I expect to find that experienced editors are much more compliant with the ODP Guidelines (http://dmoz.org/guidelines.html) than newer or less experienced ones.

Are the following correct? I expect experienced editors to be listed in more categories than less experienced editors. I also expect those with more experience to be listed higher in the category structure.

Are there any problems with these expectations? Are there better measures of editor experience available to the public on the ODP web site?

[kctipton]

Generally true, but there are some significant exceptions.

[Quadrille]

Don't forget ODP uses volunteers, not employees; this means that levels of ambition, commitment and altruism will all vary widely between individuals.

You'll see a wider variety of behaviour / responsibility / seniority in experienced editors than you might expect in a paid workforce.

And I'm sure there's other variables to undermine expectations!

[ettore]

>>I expect to find that experienced editors are much more compliant with the ODP Guidelines

Generally true, with a few exceptions

>>I expect experienced editors to be listed in more categories than less experienced editors

Not always true. Category collecting is not encouraged, and since editors have access to all subcategories of the one where they are listed editor, it is not uncommon that an editor resigns from the subcats when h/she is accepted in a higher level one

>>I also expect those with more experience to be listed higher in the category structure

Somewhat true, but with several significant exceptions due to the available time, commitment, and expectations.

Basically, what kctipton said.

[kctipton]

As the guidelines evolve, newer editors may actually be more compliant (but not always).

[furl]

Thanks for the insight OD editors. The exceptions make the game fun. In ranking editors, I only tested the categories where their names were listed. I didn’t look at the Adult or the non-English categories. This is still a work in progress with many other limitations. All this is based on stats, so your mileage may vary.

I can’t measure ambition and altruism. As far as commitment goes, I can infer results. Statistically, compliance is higher than would be expected based upon sample size. I can’t directly answer Kctipton’s question. Editors with their names on 3-5 categories were the most compliant. Height in the category tree didn’t amount to beans. I tested for systematic non-compliance in the main categories individually. The program is tons better at catching really sneaky, smart rats than dumb mice. This is a major bug. "Kids and Teens" is the only main category significantly shaped by systematic non-compliance.

[Quadrille]

Tell us more, do!!

I'm interested in how you measured compliance; as the guidelines are quite large and - like most guidelines - often open to interpretation, I'm interested in what you selected to maximise your objectivity.

I'm not at all surprised that Kidz'n'Teenz showed up systematic non-compliance; remember that local guidelines exist, mostly built on the main guidelines - K&T has much more autonomy than other areas you've considered, indeed, it might be considered a distinct entity. Interestingly, the only other areas with a significant variation due to 'local guidelines' are Adult and World, which you've excluded.

Depending on your aims, might be best to exclude K&T also.

How do we get to see the whole report? :)

[kctipton]

Kids and Teens has its own guidelines. It's really _separate_ from ODP and should be judged that way.

Your best judgement would come from examining individual editors' logs rather than categories in which they are listed. I can go and mess up lots of categories, none of which have my name on them :) Unfortunately for you, you can't see the editors' logs.

[furl]

>>>> Unfortunately for you, you can't see the editors' logs.
There’s just not enough beer on earth to get me to look at those logs. My goal is to test the software, not the ODP. Hopefully, the software is spotting trends that humans might not catch. The tests for Compliance and Systematic Non-Compliance are apples and oranges. The Compliance test looked at individual editors and no sub-categories. Compliance used a natural language processor to generate rules from the guidelines.

>>>>> I can go and mess up lots of categories

Do you wish me to post your Compliance rank? I’m teasing. Systematic Non-Compliance looked for patterns in the treatments of individual sites. It didn’t use the guidelines. The more systematic the treatment, the more likely to fire the alarm. I didn’t try to snag bad eggs because of the subcategory deal.

The treatments of sites in "Kids and Teens" were only compared with sites in "Kids and Teens". There’s too much variance to test the main categories as a group. Subdividing the main categories would enhance results.

In "Kids and Teens", a small number of sites skew the results. I looked up what the program red flagged. Web pages from one site are given special treatment in several related branches. There’s jolly fun with the descriptions. Hard not to notice that some of the descriptions and URLs contain an editor’s handle. With those pages removed, there is about as much noise in KT as in the others. I also found the same boil a few places outside KT. To an outsider it looks bad, but maybe that site just has fantastic luck. The big picture really doesn’t look too bad at all though.

>>>> How do we get to see the whole report?
I’m not writing a report on the ODP. It would make me angry to find my data misrepresented. A fair interpretation speaks well for the ODP. It is not the Wild West that I imagined when I first posted here. That’s good for the Open Directory but not for me. Thanks.