My understanding of this is that it's ok to scrape if you have permission by the original poster.
rogerd
4:27 pm on Apr 16, 2004 (gmt 0)
Very interesting. It seems that the robot restrictions (or lack thereof) were part of the thought process. This raises further issues if you want a site to be indexed in search engines but not scraped by third parties - how specific do your robots.txt and/or meta tag files have to be? Clearly, if you are set up to ban all robots, the "keep out" sign is clear. But what if a scraper uses an unfamiliar user agent? Or masquerades as a standard browser? Did they violate your robot instructions, or since the instructions didn't apply to their user agent were they irrelevant?
