Perhaps soon there will be lawmaking bodies writing laws that dictate a standard that says "You may only send a non-human agent to a website if that website embeds an invitation"? Might work.
Robots.archive? Your agent may visit and archive our content but only if you do not derive a direct or indirect profit from the archived material? Okay, that knocks out Google.
Robots.search? Your agent may visit and archive out content but only if you . . .
The laws will come.