SteveWh - 8:59 am on Sep 18, 2011 (gmt 0)
To clarify, the sequence of events that I hypothesized was:
The AV detects, on the PC, that an outgoing http request is about to be sent out (a page request). It blocks that request temporarily while the AV does 2 things:
1) While the request is blocked, it compares the URL against the list of dangerous sites. If it is known-dangerous, it leaves the request blocked and puts up a warning page in the browser instead, "This is a dangerous site." Your browser is never allowed to send the outgoing http request to the dangerous site.
2) If the site is not already in the dangerous list (and while your http request is still blocked), the URL is sent to the Trend server for a second check. The Trend bot fetches the URL and scans the result for malware. If the data is clean, Trend sends an "Ok" message back to the user's AV program. And the AV allows the browser to proceed with sending out its http request for the URL.
Then, when the page is received and stored on HD in the browser cache, it's scanned again (!) for viruses by the AV scanner.
Though I doubt they'd be able to explain the lack of a transparent UA name like "AntiVirusBot" so you know what you're dealing with.
I think they were just being sneaky, especially so that sites that distribute malware on purpose wouldn't know who their bots really are and block them. Revealing that "this is an AV scan" in the user-agent string would be an invitation to be blocked.
There are lots of these AV bots around. I don't think I've seen any of them identify themselves in the UA string, and for many, the IP doesn't even trace to the name of the company that is doing the scan.