wilderness - 4:47 am on Sep 18, 2011 (gmt 0)
The Trend bots, being part of an antivirus program's machinery -- rather than a web crawler -- are not going to read robots.txt or obey its instructions. Any URL that their user is able to visit is fair game for them to fetch and analyze for malware. On the other hand, they're not crawling it for the purpose of putting it into a public web index.
The moral of the story for a webmaster is that if you have pages or scripts that you want to be truly secret and inaccessible, you must make them physically impossible to access. Either put them in a password protected directory or apply an IP test so that the request is denied for all IPs other than yours.
The real "moral of the story" is that webmasters must choose very carefully about what Third party IP ranges, Third party bots and software's they allow access.
Third party visitors (to be cordial) are not restricted to AV software's alone, rather include filtering software's, translators, universities and a variety of other capabilities, all of which feel that any website is fair game.