Here is the NS lookup
We were just discussing the graphics on their web site [antarcti.ca].
Bjaaland/0.5 ODP-stats (firstname.lastname@example.org) libwww-perl/5.44
It is a LWP perl bot. But does anyone know if this is the (or one of the) DMOZ link verification spiders?
Absolutely right, littleman. [info.webcrawler.com]
I thought it was, but what made me do a double take was the place it is coming out of.
Has anyone else been visited by Robozilla/1.0
It may be another DMOZ link verification spider. Only my indexed page was visited and nothing else.
also visited with the same referring URL and User-agent.
I just got hit with over 350 attempts at bogus pages and directories within minutes by this guy ( 18.104.22.168 ). What a pain! I got 350+ emails with telling me someone got 404 errors on my site. What a mess!
At first I thought it was checking for links, for it took every document with a GET request, and then every image on the site with a HEAD request.
But then I went to antarcti.ca, and went to the demo here:
Basically it is a visual ODP. Quite impressive I think. But you'll need to be using something faster than 56K unless you're really patient.
You can do it 2D or 3D. Really cool.
These guys hit my site pretty hard, and due to my configuration problem (with my log program), I didn't see their robots.txt request until *after* I emailed them compaining loudly about their bad manners... how dare they not request robots.txt, and therefore send their spider barreling through directories it had no business in, etc., etc.
So I got an email back from a tech support guy, who (after I realized my error and apologized) took the time to test and re-test my robots.txt with me, until it was working properly. Their customer response and service for their internet spidering is absolutely impeccable.
Although I think the actual major thrust of the corporation is to develop search/indexing solutions for large corporate intranets...
Robozilla is the test spider for DMOZ. It goes round seeing if the site is a 404 or not.
The Bjaaland one is just a link verification spider for antarcti. Why they don't just download the rdf dump again is a mystery...
no, I don't think Bjaaland is *only* link verification. It spidered my *entire* site on it's first run through.