Forum Moderators: phranque
[psychedelix.com...]
For myself at least the Fetch API Request never reads robots.txt, they don't go away if you generate 403 and each example I attempt to track back never seems to lead to any obvious reason why they would need to spider the page.
The ISA Server theory is supported by the geographically distributed nature of the requests as well as the request sources (nearly always non-isp corporate owned), not to mention that if you view the raw request most of them go through machines whose names include "ISA".
- Tony