ytswy - 7:09 pm on Mar 18, 2007 (gmt 0)

If there's no robots.txt, I think it's also valid check the index page for meta tags that control crawler activity as well, if these don't exist either, which don't on her site, the crawler should keep out. Basically, the crawler wasn't invited in but it also wasn't blocked, so there's no defined permission granted either way. Politeness alone would dictate to err on the side of caution until the webmaster figures out they have no traffic and learn to install a robots.txt file with permissions and explicitly grant spiders access.

Goodbye Google et al. It was nice while it lasted, but unfortunately we've decided that copyright owners interests trump any benefits you've provided the web over the last ten years.

I come back again to the fact the copyright is not an absolute right. It is an incentive to create, based upon the reasoning that society at large gains more in terms of the additional content created than it loses from enforcing monopolies. I simply fail to see why any sane society would want to enforce these draconian measures on crawlers, especially when a widely accepted opt-out system already exists with robots.txt.