joined:Nov 5, 2010
Short version: I would like to disable Google's indexing during certain days/times. I am also concerned about changing my robots.txt file and what the long term effects are (do they not come back?)
My company hosts monthly sales on our website where our traffic increases dramatically. Yesterday, the need for heavy load testing an optimization was pushed to the forefront during the perfect storm of heavy use and a Google crawl.
The fact that in just a couple hours, during our heaviest period of use, Google had downloaded well over a gigabyte of data (not to mention the stress on the SQL server), this was more than enough to push the system over the tipping point to horrible performance (1000% typical load time).
Does anyone have experience with similar problems? Were you able to find acceptable solutions while still maintaining good search results? We are currently top 3 on all our important search terms and phrases and I would hate to lose that. But if our site doesn't work, that is worse.