Odp

watch your logs after you subm... er, suggested a site.

[bull]

207.200.81.151 - - [09/Jan/2005:01:00:00 +0100] "HEAD / HTTP/1.1" 206 0 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

The bot from dirt05.netscape.com == research.dmoz.org formerly identified itself as "-". Now it adapts the behavior of Spambots and uses a faked MSIE UA String.

[volatilegx]

Nice catch :)

[choster]

Just to clarify, the various bots on research.dmoz.org are, well, used for research. In other words, the editor running a bot against newly suggested sites is not necessarily the same one handling public submissions there. In fact, I'd wager most additions to the directory are still made without the use of automated tools, except for culling spam.

[pmkpmk]

As far as I understood, the ODP index is crawled by it's own bot in order to find sites which went offline.

When an editor (manually) edits a site, then you will see "editors.dmoz.org" as the referrer. That's the best telltale sign that an editor has actually taken a look at your site.

[bull]

choster,

my above statement is correct as I reported the HEAD requested by a "-" User-agent in a large german SEO forum on November, 21, 2004. Also this request was made few hours after a site was suggested for ODP inclusion, exactly like the one I reported above. I even started a thread in the german forum of no-no-zone.
It is the same bot.

[choster]

I did not say you were making it up. But I am saying authoritatively that the submission/review/acceptance process is independent of any bot. There are bots on research.dmoz.org which, for instance, run Google queries, or which may otherwise land on a site whether or not it has been submitted. The automated link checker ("Robozilla") only runs against sites already in the directory. And someone who "touches" a suggested site using a research bot is not necessarily even opening it, s/he could be running an automated sorting or spam deletion process for instance.

[bull]

I said that
1. in Nov. 2004 few hours after submission a HEAD request was made to a suggested domain with the UA string "-"
2. in Jan. 2005 few hours after submission a HEAD request was made to another suggested domain with the UA string "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
3. that the two bots are the same, because they show the same behavior.

But I am saying authoritatively that the submission/review/acceptance process is independent of any bot.

Please re-read my posts. I never wrote that the bot had anything to do with the review etc. process. When speaking with DMOZ people I sometimes feel like in a special school ;)