SteveWh - 4:09 am on Sep 18, 2011 (gmt 0)
I used Trend Micro Internet Security for about 5 years.
It seemed that the software would send a URL that I typed into my browser address bar back to Trend even before my browser could fetch the page. I used to see requests from the Trend IPs (150.70. is one set of them) showing up in my logs even before my own request (if I recall correctly), and definitely at other times just a second or two afterwards, even for files that were secret, only on my server for a few seconds, and that I only requested once and then deleted.
The software also maintained a list on my hard drive of all the URLs I visited (whether by typing into the address bar or by clicking links). That list was sent to Trend with each update. In that case, they'd crawl those URLs sometime later in the day.
The Trend bots, being part of an antivirus program's machinery -- rather than a web crawler -- are not going to read robots.txt or obey its instructions. Any URL that their user is able to visit is fair game for them to fetch and analyze for malware. On the other hand, they're not crawling it for the purpose of putting it into a public web index.
The moral of the story for a webmaster is that if you have pages or scripts that you want to be truly secret and inaccessible, you must make them physically impossible to access. Either put them in a password protected directory or apply an IP test so that the request is denied for all IPs other than yours.