fumanchu - 8:13 pm on May 8, 2013 (gmt 0) [edited by: incrediBILL at 10:29 pm (utc) on May 8, 2013]
I read a great post by Incredibill in another thread (http://www.webmasterworld.com/google/4081021-2-30.htm), and I wanted to share my experience with you.
I have a popular blog that has been battered by scraper bots from foreign countries. The other day I installed a free plugin for WordPress, then upgraded it to the paid API so I could block countries.
I blocked all countries except for UK, US, Australia and Canada (my target markets). What were the results of this?
1. My server load dropped HUGELY.
2. My adsense revenue more than doubled the first day, and is way up again today.
3. My CPC is way up since I got rid of all the low paying clicks from countries like India, China, etc.
4. The scraper bot plague from china, russia, etc. has ended.
5. My blog loads very fast now and has been very stable.
6. I took Bill's advice and used the <meta name="robots" content="noarchive"> to get my content out of google's cache. I can confirm that it took a few hours and it was gone.
7. I blocked the IA archiver thing in robots.txt with this:
Some tips about how I'm using this (aside from the country blocking):
1. Turn off the Live Traffic feature until you actually need to use as it will use up more server resources if you just leave it on (not that it's that bad, but if you aren't using it why keep it on?).
2. I set a very aggressive control for crawlers (lets you set different rules for crawlers and humans). If they violate the server rules, they are automatically blocked for one month. Yes, a month. Why? Most of these stupid crawlers add little or no value whatsoever to my sites. So I don't care if they can't access them.
3. Verified googlebots have unrestricted access. So the server rules for the other crawlers don't affect them.
4. Fake googlebots get blocked automatically.
Anyway, I'm a happy camper right now. The scraper bots have been stressing me out until now. I really didn't think of blocking foreign countries until very recently. At first I was afraid I'd lose a lot of money, but after looking at my adsense performance reports, the amount was negligible compared to my target markets.
I have to admit that I was very surprised that I ended up making more money, with less traffic. I was scared that my adsense earnings would drop, but they didn't.
I have actually moving to a cheaper VPS since the load on the server is so much lower, I don't need the higher cost one I was paying for. So the blocking is going to save me about $25 per month in hosting costs.
I hope this post helps some of you out there who might have had similar problems.
BTW, I'm not a server guy, so fiddling with server settings and filters isn't something in my realm of skill sets, but the plugin worked really well for me, and it was easy to use. I know that some of you can do the server voodoo stuff instead of using the plugin, but I can't.
And thanks again to Bill for his awesome post and encouragement to focus on target markets and not be afraid to block foreign countries. Thanks Bill! :)
[edited by: incrediBILL at 10:29 pm (utc) on May 8, 2013]