homepage Welcome to WebmasterWorld Guest from 54.205.205.47
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
asterias/2.0
Looks for streaming media files...
pendanticist




msg:1529437
 6:52 am on Jan 22, 2004 (gmt 0)

Found a new one perusing my site a little bit ago and figured I'd pass it along, since it hasn't been mentioned here before.

http://www.singingfish.com/help/spider.html

206.253.222.*** - - [21/Jan/2004:13:21:57 -0800] "GET / HTTP/1.1" 200 20402 "-" "asterias/2.0"
206.253.222.*** - - [21/Jan/2004:13:21:58 -0800] "GET /robots.txt HTTP/1.1" 200 1524 "-" "asterias/2.0"

Reading thru these results [google.com] seems to indicate this critter has no business on my site.

I'm thinking:

User-agent: asterias
Disallow: /

Would not be as good as:

RewriteCond %{HTTP_USER_AGENT} asterias [NC,OR]

I believe I have that correct.

I don't much care if it asks for and obeys robots.txt or not, I simply don't have any files it'd need for that engine and can NOT see allowing it.

 

bcolflesh




msg:1529438
 5:42 pm on Jan 23, 2004 (gmt 0)

[webmasterworld.com...]

[webmasterworld.com...]

pendanticist




msg:1529439
 9:52 pm on Jan 23, 2004 (gmt 0)

The question was in the last statement, ( albiet ambiguously ):

I'm thinking:

User-agent: asterias
Disallow: /

Would not be as good as:

RewriteCond %{HTTP_USER_AGENT} asterias [NC,OR]

I believe I have that correct?

Question mark added this time.

bakedjake




msg:1529440
 9:54 pm on Jan 23, 2004 (gmt 0)

p - that's correct, especically if it's weird about not obeying your robots.txt. Their bot has been evil to a couple of our sites in the past.

Note: You may want to revisit this rule in 6 months. Singingfish could become a semi-major player. It's been getting a lot of hype.

If you don't have multimedia content, however, it's probably nothing to worry about.

pendanticist




msg:1529441
 10:07 pm on Jan 23, 2004 (gmt 0)

There is nothing on my site they'd ever want and I'd just as soon ban them if for no other reason, than to save bandwidth. :)

Thanks.

keyplyr




msg:1529442
 11:43 pm on Feb 1, 2004 (gmt 0)

Just had asterias; 2.0 hit my site. It did not obey robots.txt, hell... it didn't even request it.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved