-- Search Engine Spider and User Agent Identification
---- 150.70. revisited
SteveWh - 6:18 am on Sep 18, 2011 (gmt 0)
I only remember them fetching main files like .htm, .html, .zip, and not .css, .js, but if they are fetching those others now, it would be a reasonable escalation because malware is being stored in those nowadays, too.
The Regular Expression Is Your Friend
But what kind of security are you providing when you don't even look at a page until more than an hour after the human visit that triggered your inspection? Anything can happen in an hour.
Like any AV, it inspects the file after the user's browser has received it and written it to hard disk in the browser cache. That's normal AV activity.
This pre-fetch behavior is more pro-active than that if it really did intercept the browser request and fetch the page *before* the user does. In other words, it's like saying, "Wait a minute, I'm not going to let you get that page until I've checked it out FIRST." Then the Trend bot fetches the page, examines it, and if it's ok, sends a message back to the user's AV, which allows their browser request to proceed.
I'm not sure it was doing that but did suspect it.
Also, Trend, like other AV companies, maintains its own list of unsafe websites. If you try to go to one, it blocks your request even before you can get a Google/Firefox Safe Browsing message or Internet Explorer warning. Some of this crawling could be in support of keeping their list updated.