Forum Moderators: open
First seen 2-22-08 coming from "Perspektiv Bredband AB" ip range 81.186.240.0 - 81.186.255.255
It started by taking only RSS documents, and today (3/3/2008) it tried to crawl the links in the RSS Documents.
I allow bots to read my RSS Feeds but do not allow bots to crawl any further unless they read Robots.txt and respect it. This one did not read Robots.txt nor has it ever. And the website does not contain any useful information what so ever.
So I am wondering what everyone else thinks of it?
Ocean
[edited by: engine at 3:52 pm (utc) on Mar. 5, 2008]
Of course the difference here is you don't have advertising on your pages and I do so I need to make sure people click through to the site and don't just see the scraped content elsewhere.