Tarantula

[wilderness]

robots and main page.
No images

67.188.79.xx - - [12/Oct/2005:14:57:59 -0700] "GET /robots.txt HTTP/1.1" 200
3534 "-" "Tarantula/0.1 (compatible; Mozilla 4.0; MSIE 5.5;
[e-mail])"

[Stefan]

Saw it today. Robots.txt and one of our more popular pages. At least it's polite.

[wilderness]

Stefan,
Tarantula is a pretty name for Spider ;)
With the exception of Lycos!

I'm unable to recall a "creditable" bot using the term in their UA. (Maybe it's a frame-of-mind.)

Why would a bot desire to associate their name and brand with a name/image that has a bad reputation?

Course, I guess anything's possible.

Don

[Staffa]

I had the same visit and the same reaction

Why would a bot desire to associate their name and brand with a name/image that has a bad reputation?

but to the owner it must have come naturally : mark_taran(tula)

[Stefan]

Thanks Don. It didn't even occur to me. Funny thing is that Araneae are something I have an interest in (other than wrt webmastering).

This would probably be better as a new thread, but has anyone else noticed "Rufus"? I haven't found much on the net.

[Leosghost]

One of the best basic html tutors /editors for newbies is arachnophilia ...even "W" could build a site with it!

[GaryK]

This bot visited my site recently. It didn't read robots.txt until after it had started its crawl. That's my only complaint about it but it's enough for me to ban it.

[GaryK]

I apologize for double posting but it was too late to edit my previous message. Does anyone know the URL for this crawler? It would have been nice if it were included in the user agent!

[wilderness]

Gary,
Whether this is the same software, I'm not sure?

[topshareware.com...]

[edited by: volatilegx at 9:15 pm (utc) on Nov. 4, 2005]
[edit reason] fixed broken URL [/edit]

[GaryK]

Don, that appears to be software for searching Amazon.com. I wonder why someone would crawl hobby sites and my main site with it? Then again I didn't actually download the software. :)