After that, they all started crawling my site, on an average half a dozen pages per minute, probably well past midnight, where my current log file ends. Looks like each one of those 16 robots fetched every single HTML document from my site seperately.
I wonder if they will now rank my site cumulatively in exchange? ;)
Yesterday, 020202, 12 individual FAST servers (66.77.73.*) made 12 visits to one of my little sites and each of the 12 FAST servers went to nearly half of my entire site. Is this normal behaviour for FAST or should I put in a command in robots.txt?
One of our engineers was testing the performance of a new feature using a test-cluster of 16 crawler machines. Unfortunately, he forgot to configure the crawlers to divide the work between them, so they all acted independetly and effectively all crawler were crawling all webservers. We got some feedback on this quite quickly and the configuration error was then corrected.
Sorry for the convenience and thanks for the feedback!