dumb spider: asterias/2.0

coho.singingfish.com: asterias/2.0

[skirril]

Indexes Multimedia Documents (www.singingfish.com). Cannot cope with forms. Honors robots.txt

[littleman]

Thanks for the headsup!

[Brett_Tabke]

But are they sending any referrals? From who?

[singingfish.com]

They look like bandwidth sponges of the worst order to me. Direct indexing of mp3/multimedia on your site is not something that is generally beneficial to a site.

[skirril]

All "multimedia content" I have on my site are
jpeg images. Since I don't know of an effective
means to 'search for images' other than by name
(which is completely useless), I disallow all
image files to all robots:

User-agent: *
Disallow: /img

So I dont get any referrals from Asterias.

[Son_House]

Hi All

I'm new to spider identification so please be gentle if I'm wrong here. I banned asterias in my robots txt last week and yesterday it came back. This time it also showed up as Java1.3.0 Two requests as asterias and one as Java1.3.0 Is that also a spider and should I add Java1.3.0 to my robots txt? If yes to those two questions, what are the chances of a friendly spider also showing up as Java1.3.0?

63.251.10.136 - - [28/Feb/2001:15:55:58 -0500] "GET /robots.txt HTTP/1.1" 200 1582 "-" "asterias/2.0"
63.251.10.136 - - [28/Feb/2001:15:55:58 -0500] "GET / HTTP/1.1" 200 6543 "-" "Java1.3.0"
63.251.10.136 - - [28/Feb/2001:15:56:04 -0500] "GET / HTTP/1.1" 200 6524 "-" "asterias/2.0"

[mivox]

Welcome to WebmasterWorld, Son_House!

If you notice, all three visits from this spider come from the same IP address. You can ban visitors by IP address as well as UserAgent, so look into banning asterias that way instead. Here's [shat.net] some information about how to do that.

singingfish.com also gives an email address for asking them to exclude you from any future visits:

"If you wish to stop Asterias® from crawling your site, simply click here to send an email to our Operations Team ([e-mail]). Please include the name of the site you wish to exclude."

[Son_House]

Hi mivox

Thank you for the warm welcome!

Thank you also for the link and the other info about singingfish.com. I found it very helpful.

[mivox]

No problem... until that message, I didn't really know how to ban visitors by IP either, so when I looked up the info for you, I learned something new as well! :)