Forum Moderators: open

Message Too Old, No Replies

New RSS Reader - Spider?

Asa/1.0.1

         

Ocean10000

3:57 am on Mar 4, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



User-Agent
Mozilla/5.0 (compatible; ASA/1.0.1 http://www.zine.se/)

First seen 2-22-08 coming from "Perspektiv Bredband AB" ip range 81.186.240.0 - 81.186.255.255

It started by taking only RSS documents, and today (3/3/2008) it tried to crawl the links in the RSS Documents.

I allow bots to read my RSS Feeds but do not allow bots to crawl any further unless they read Robots.txt and respect it. This one did not read Robots.txt nor has it ever. And the website does not contain any useful information what so ever.

So I am wondering what everyone else thinks of it?

Ocean

[edited by: engine at 3:52 pm (utc) on Mar. 5, 2008]

incrediBILL

1:07 am on Mar 17, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Most RSS feed readers don't consider themselves crawlers therefore feel entitled to take anything you link via the RSS feed which is why I block them from reading anything but the RSS feed itself, regardless of robots.txt access.

Of course the difference here is you don't have advertising on your pages and I do so I need to make sure people click through to the site and don't just see the scraped content elsewhere.