Radian6 CommentReader

reads robots.txt - but does it obey it?

[caribguy]

142.166.170.nnn - - [04/Aug/2008:20:33:34 -0500] "GET /robots.txt HTTP/1.1" 200 926 "-" "R6_CommentReader(www.radian6.com/crawler)"
142.166.170.nnn - - [04/Aug/2008:20:33:34 -0500] "GET /example/page HTTP/1.1" 200 81329 "-" "R6_CommentReader(www.radian6.com/crawler)"

No usable info on whether it honors robots.txt on the /crawler page. Apparently they help companies find and listen to conversations about their brands. - something I could care less about...

[edited by: incrediBILL at 7:13 am (utc) on Aug. 5, 2008]
[edit reason] Obscured IPs [/edit]

[incrediBILL]

When I saw them it was 142.166.3.nnn so they're moving around.

They don't bother with robots.txt because they're a feed reader that attempts to step off the feed and snag the article.

[wilderness]

This Maritimes provider has long been a home for "some" rogues.

Years ago there was a regular harvester that was based in one of the providers many ranges.