Forum Moderators: open
simulates a browserHow lifelike is the simulation? Do supporting files get requested suspiciously fast*, after a reasonable delay, after an unreasonable delay, or not at all?
How lifelike is the simulation?The operator can edit the UA string to say anything. You've likely been hit without knowing it. The requests could be innocent (server admin using this script to browse sites) or it could be scraping (as jmccormac mentioned.) The requests can be controlled to GET all files on a document (like a normal browser does) or just set to request specific file types (scraping behavior.)