Bookmarks of the DMOZ-editors

a question about the robots.txt

[kosmodrom]

When i fetch the robots.txt at dmoz it shows me

User-agent: *
Disallow: /cgi-bin/
Disallow: /editors/
Disallow: /Bookmarks/
Allow: /Bookmarks/A/autumn/
Allow: /Bookmarks/C/cptginyu/
Allow: /Bookmarks/D/dole/
Allow: /Bookmarks/L/lexib/
Allow: /Bookmarks/S/skrenta/
Allow: /Bookmarks/T/truel/

So, if i get this right: all bookmark-pages of the editors won't be crawled, except six.

My question is: why are these six bookmark-pages allowed to crawled?

Who decides which bookmaks are crawled and which ones not? The editor himself?

Can an editor allow or dissalow his/her bookmarks to be crawled?

[Mike_Mackin]

>Who decides which bookmaks are crawled and which ones not?

It seems that ROOT editors decide.

[tedster]

I can't say what they're trying to do, but that isn't standard syntax for robots.txt. It only uses "Disallow", not "Allow".

the line "Disallow: /Bookmarks/" bans ALL files in the Bookmarks directory, and the "Allow" lines will not be effective in getting a few exceptions to be crawled, IMO.

[andreasfriedrich]

tedster is right.

The only two headers recognized in a robots.txt file are User-agent and Disallow. All other headers are ignored. See [robotstxt.org...] for more.

[ettore]

Aside from the Allow/Disallow issue, just answering the original question:

>> why are these six bookmark-pages allowed to crawled?

autumn = ODP Staff sw engineer
cptginyu = ODP former Staff sw engineer, still helping autumn and working in Netscape/AOL/TW
dole = Bryn Dole is Principal SW Engineer at Netscape/AOL/TW, responsible for the Open Directory Search and working on AOL Music Search and AOL Shopping Search.
lexib = Root editor, Netscape Staff
skrenta = Rich Skrenta is the founder of the ODP, the one who invented the whole thing.
truel = Bob Truel, co-founder of the ODP

>> Who decides which bookmaks are crawled and which ones not?

Staff decides

>> The editor himself?

No

>> Can an editor allow or dissalow his/her bookmarks to be crawled?

No

[fathom]

The advanatges of being "in the know club", or not!

Too bad someone didn't understand what they were doing.