Another Weird Spider?

[steven]

209.17.159.193 - - [25/Oct/2000:18:53:49 -0600] "GET /robots.txt HTTP/1.0" 200 649 "-" "Bjaaland/0.5 ODP-stats ([e-mail]) libwww-perl/5.44"

Which SE does this spider belong to?

[littleman]

Here is the NS lookup
Name: lb1.antarcti.ca
Address: 209.17.159.193
We were just discussing the graphics on their web site [antarcti.ca].

Bjaaland/0.5 ODP-stats ([e-mail]) libwww-perl/5.44

It is a LWP perl bot. But does anyone know if this is the (or one of the) DMOZ link verification spiders?

[bartek]

Absolutely right, littleman. [info.webcrawler.com]

[littleman]

Thanks Bartek,
I thought it was, but what made me do a double take was the place it is coming out of.

[Gorufu]

Has anyone else been visited by Robozilla/1.0

It may be another DMOZ link verification spider. Only my indexed page was visited and nothing else.

Server: h-206-222-248-44.netscape.com
Referrer: "http://directory.mozilla.org"
UA: Robozilla/1.0

h-208-12-52-94.netscape.com
also visited with the same referring URL and User-agent.

[mikeneal]

I just got hit with over 350 attempts at bogus pages and directories within minutes by this guy ( 209.17.181.33 ). What a pain! I got 350+ emails with telling me someone got 404 errors on my site. What a mess!

[bobriggs]

209.17.181.33
Bjaaland/0.6 ([e-mail])

At first I thought it was checking for links, for it took every document with a GET request, and then every image on the site with a HEAD request.

But then I went to antarcti.ca, and went to the demo here:
[maps.map.net...]

Basically it is a visual ODP. Quite impressive I think. But you'll need to be using something faster than 56K unless you're really patient.

You can do it 2D or 3D. Really cool.

[mivox]

These guys hit my site pretty hard, and due to my configuration problem (with my log program), I didn't see their robots.txt request until *after* I emailed them compaining loudly about their bad manners... how dare they not request robots.txt, and therefore send their spider barreling through directories it had no business in, etc., etc.

So I got an email back from a tech support guy, who (after I realized my error and apologized) took the time to test and re-test my robots.txt with me, until it was working properly. Their customer response and service for their internet spidering is absolutely impeccable.

Although I think the actual major thrust of the corporation is to develop search/indexing solutions for large corporate intranets...

[backus]

Robozilla is the test spider for DMOZ. It goes round seeing if the site is a 404 or not.

[Brett_Tabke]

The Bjaaland one is just a link verification spider for antarcti. Why they don't just download the rdf dump again is a mystery...

[mivox]

no, I don't think Bjaaland is *only* link verification. It spidered my *entire* site on it's first run through.