Welcome to WebmasterWorld Guest from 22.214.171.124
some of those are Bing
There is no law against it, after all.
And they don't necessarily have "Bot" or "bot" in the UA
pages I did not even know existed
If the robots choose to ignore his instructions and access the files anyway, then perhaps they fall foul of that act.
also deny directives and RewriteRules detecting IPs and UAs.
Preview isn't a robot. You and I may think it is, but g### has decided otherwise. After a few weeks of "Which part of 'Disallow: /piwik' did you not understand?" I caved in and went to a more robust