DutchTheWiz - 12:38 pm on Aug 29, 2013 (gmt 0)
I am just a developer who was in charge for writing code to download files up to 10GB - thousands of them. To do this I had to understand behaviors and distinguish between good and bad requests and somehow avoid downloads when possible(GET, HEAD, Last Modified, etc).
In the other hand it is desired or even required that these files are public and available to any friendly customers.
My Network group is in charge for tuning balancer, so I am hoping that they will make it work soon and that will be able to block malicious IPs. I do not need to mention how malicious requests slowing down downloads.
And yes lucy it is "google-xrawler"