I run a e-commerce website with a 100GB lucene index and a 30GB Mysql database. I have a load balancer and two E5-2690 servers with 32GB of RAM, 1000Mbps ethernet, and a 400GB Kingston KG-S284X SSD drive. So a really beefy setup, in my opinion.
I'm running a MVC website written in c# and .NET 4.5 on IIS 8.0, windows server 2012.
Around Christmas, I noticed something strange. During peak traffic times my YOY sales growth would slow. Then, this year, I noticed that on days when a couple of well mannered rogue scrapers where on my site, sales growth would slow as well. Some of these scrapers are browser based and load all page resources.
Any ideas on where I should look?
[edited by: jskrewson at 2:58 pm (utc) on Apr 17, 2014] Added edit by engine at member request