Welcome to WebmasterWorld Guest from 50.16.24.12

Forum Moderators: Ocean10000 & incrediBILL

Another Weird Spider?

   
7:45 pm on Oct 26, 2000 (gmt 0)

10+ Year Member



209.17.159.193 - - [25/Oct/2000:18:53:49 -0600] "GET /robots.txt HTTP/1.0" 200 649 "-" "Bjaaland/0.5 ODP-stats (bjaaland@antarcti.ca) libwww-perl/5.44"

Which SE does this spider belong to?

5:15 am on Oct 27, 2000 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Here is the NS lookup
Name: lb1.antarcti.ca
Address: 209.17.159.193
We were just discussing the graphics on their web site [antarcti.ca].

Bjaaland/0.5 ODP-stats (bjaaland@antarcti.ca) libwww-perl/5.44

It is a LWP perl bot. But does anyone know if this is the (or one of the) DMOZ link verification spiders?

5:48 am on Oct 27, 2000 (gmt 0)

10+ Year Member



Absolutely right, littleman. [info.webcrawler.com]
6:34 am on Oct 27, 2000 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Thanks Bartek,
I thought it was, but what made me do a double take was the place it is coming out of.
11:30 am on Oct 27, 2000 (gmt 0)

10+ Year Member



Has anyone else been visited by Robozilla/1.0

It may be another DMOZ link verification spider. Only my indexed page was visited and nothing else.

Server: h-206-222-248-44.netscape.com
Referrer: "http://directory.mozilla.org"
UA: Robozilla/1.0

h-208-12-52-94.netscape.com
also visited with the same referring URL and User-agent.

12:14 pm on Mar 31, 2001 (gmt 0)

10+ Year Member



I just got hit with over 350 attempts at bogus pages and directories within minutes by this guy ( 209.17.181.33 ). What a pain! I got 350+ emails with telling me someone got 404 errors on my site. What a mess!
2:47 am on May 11, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



209.17.181.33
Bjaaland/0.6 (bjaaland@antarcti.ca)

At first I thought it was checking for links, for it took every document with a GET request, and then every image on the site with a HEAD request.

But then I went to antarcti.ca, and went to the demo here:
[maps.map.net...]

Basically it is a visual ODP. Quite impressive I think. But you'll need to be using something faster than 56K unless you're really patient.

You can do it 2D or 3D. Really cool.

7:35 pm on May 11, 2001 (gmt 0)

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member



These guys hit my site pretty hard, and due to my configuration problem (with my log program), I didn't see their robots.txt request until *after* I emailed them compaining loudly about their bad manners... how dare they not request robots.txt, and therefore send their spider barreling through directories it had no business in, etc., etc.

So I got an email back from a tech support guy, who (after I realized my error and apologized) took the time to test and re-test my robots.txt with me, until it was working properly. Their customer response and service for their internet spidering is absolutely impeccable.

Although I think the actual major thrust of the corporation is to develop search/indexing solutions for large corporate intranets...

1:22 pm on May 14, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Robozilla is the test spider for DMOZ. It goes round seeing if the site is a 404 or not.
5:39 pm on May 14, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The Bjaaland one is just a link verification spider for antarcti. Why they don't just download the rdf dump again is a mystery...
6:05 pm on May 14, 2001 (gmt 0)

WebmasterWorld Senior Member mivox is a WebmasterWorld Top Contributor of All Time 10+ Year Member



no, I don't think Bjaaland is *only* link verification. It spidered my *entire* site on it's first run through.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month