Welcome to WebmasterWorld Guest from 22.214.171.124
Bjaaland/0.5 ODP-stats (email@example.com) libwww-perl/5.44
It is a LWP perl bot. But does anyone know if this is the (or one of the) DMOZ link verification spiders?
It may be another DMOZ link verification spider. Only my indexed page was visited and nothing else.
also visited with the same referring URL and User-agent.
At first I thought it was checking for links, for it took every document with a GET request, and then every image on the site with a HEAD request.
But then I went to antarcti.ca, and went to the demo here:
Basically it is a visual ODP. Quite impressive I think. But you'll need to be using something faster than 56K unless you're really patient.
You can do it 2D or 3D. Really cool.
So I got an email back from a tech support guy, who (after I realized my error and apologized) took the time to test and re-test my robots.txt with me, until it was working properly. Their customer response and service for their internet spidering is absolutely impeccable.
Although I think the actual major thrust of the corporation is to develop search/indexing solutions for large corporate intranets...