Welcome to WebmasterWorld Guest from 54.147.63.124

Forum Moderators: goodroi

Message Too Old, No Replies

asterias/2.0

Looks for streaming media files...

     
6:52 am on Jan 22, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
posts:1685
votes: 0


Found a new one perusing my site a little bit ago and figured I'd pass it along, since it hasn't been mentioned here before.

http://www.singingfish.com/help/spider.html

206.253.222.*** - - [21/Jan/2004:13:21:57 -0800] "GET / HTTP/1.1" 200 20402 "-" "asterias/2.0"
206.253.222.*** - - [21/Jan/2004:13:21:58 -0800] "GET /robots.txt HTTP/1.1" 200 1524 "-" "asterias/2.0"

Reading thru these results [google.com] seems to indicate this critter has no business on my site.

I'm thinking:

User-agent: asterias
Disallow: /

Would not be as good as:

RewriteCond %{HTTP_USER_AGENT} asterias [NC,OR]

I believe I have that correct.

I don't much care if it asks for and obeys robots.txt or not, I simply don't have any files it'd need for that engine and can NOT see allowing it.

5:42 pm on Jan 23, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0

9:52 pm on Jan 23, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
posts:1685
votes: 0


The question was in the last statement, ( albiet ambiguously ):

I'm thinking:

User-agent: asterias 
Disallow: /

Would not be as good as:

RewriteCond %{HTTP_USER_AGENT} asterias [NC,OR]

I believe I have that correct?

Question mark added this time.

9:54 pm on Jan 23, 2004 (gmt 0)

Administrator

WebmasterWorld Administrator bakedjake is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 8, 2003
posts:3783
votes: 2


p - that's correct, especically if it's weird about not obeying your robots.txt. Their bot has been evil to a couple of our sites in the past.

Note: You may want to revisit this rule in 6 months. Singingfish could become a semi-major player. It's been getting a lot of hype.

If you don't have multimedia content, however, it's probably nothing to worry about.

10:07 pm on Jan 23, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 27, 2002
posts:1685
votes: 0


There is nothing on my site they'd ever want and I'd just as soon ban them if for no other reason, than to save bandwidth. :)

Thanks.

11:43 pm on Feb 1, 2004 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5811
votes: 64


Just had asterias; 2.0 hit my site. It did not obey robots.txt, hell... it didn't even request it.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members