A Way to Prevent Google From Serving My Pages In India?

[Sally Stitts]

I am not a programmer.
Is there a way to prevent Google from indexing and serving my web pages in certain foreign countries, such as India, Pakistan, China and Malaysia?
Reasons -
- Reduce content theft
- Prevent AdSense ads from being served, thereby improving all my AdSense stats

I am quite popular in India and Pakistan, and I don't want to be. I feel that showing my pages over there is costing me money, additional competition and lowered ranking.

I think I saw an old post about putting something on my Apache server (rejection IP ranges?), but that you had to input a lot of data. Ideas?

[kidder]

That is an interesting idea, you can of course setup Geo targeting in GWT, I never thought about the idea of a total block but I'm sure its possible. I would be looking at the server option rather than the Google option. People will find your site in more ways than just Google. I guess a lot of comments spam etc gets outsourced to certain poorer countries so it might be an interesting experiment. I have a friend here in Australia is blocking everything he can from outside AU to stop credit card fraud on his membership site.

[Sally Stitts]

. . setup Geo targeting in GWT

I just did that, and I will try it for a week, to see if there is any improvement. But from what I have read, it may have no effect at all.

I really would like to not show up at all in these countries, but how to do this is beyond my skill level.

[tedster]

I'd say you've got to do the blocking at your server. If Google indexes a page, then it may show that page anywhere in the world and there's no way for webmasters to choose specific countries and not others.

[Sally Stitts]

If Google indexes a page, then it may show that page anywhere in the world and there's no way for webmasters to choose specific countries and not others.

That's a shame. It would really be nice, and allow us to solve several problems.

AHA!
I just found a page that will generate IP blocking code for any country, to be added to an .htaccess file. For FREE! Just enter the country, hit "Go", and there is your code.

The problem is, that there are over 400 lines of code, for India alone. Yikes. Over 100 for Pakistan. Over 1,200 for China. I was hoping for a somewhat easier, more elegant solution. I don't think it is a good idea to have an .htacess file with thousands of lines in it. Wouldn't that significantly slow down access for everybody else? Especially since Google has recently placed such a premium on speed.

[edited by: Sally_Stitts at 7:09 am (utc) on Feb 16, 2010]

[eltercerhombre]

If you want block by country you'l need to install a GeoIP DB inyour server, and then justcheck for the country code and block if needed.

However,you will be anyway showing in Google, and if they want to copy your site they can do anyway by using a proxy from a country you do not block.

But probably it's better than nothing.

[Sally Stitts]

But probably it's better than nothing.

Yeah, I'll even go for a 10% improvement.

"GeoIP DB" - I think that is the magic word I was looking for.

I will email my host about the ease of implementation of this DB. This certainly sounds like the best approach if it can be easily and reliably implemented, with no real cost. Both of which I am totally ignorant of.
EDIT - I found a site with all the perfect info. I know I can't link to it, but with some concentrated Google searching, it will pop up. I now know the buzzwords I needed, and app names, to be able to talk to my ISP with some understanding.
Two of the modules are -
- Blocking unwanted countries
- Allowing only specified countries

I think I like #2 even better. Let's see. (alphabetical, for fairness)

1-Australia
2-Canada
3-Ireland
4-New Zealand
5-UK
6-USA

Done!

[wilderness]

There's really not a simple answer to this.
Either you spend lots of time achieving the regional vieweing
OR
You pay somebody for the same goal (not foolproof and errors will occur).

Just using "white listing" and allowing the six specific country IP's you've specified would make your htaccess fairly large (in the thousands of lines), however a properly structured htacess may handle this many lines easily, absent any noticeable loss in access speed.
Just as long as the htaccess lines are NOT CPU/Server intensive.

There are multiple requirements that work unison:
1) Deny major SE's that offer bots strictly focused upon the regions you DO NOT desire.
2) DENY country refers
3) Deny translators
4) begin your IP denys with specific and complete Class A's.
5) make adjustments in your Class A's to allow access for exceptions.

Here's are some old Webmaster World threads in which I've saved links:

IP's Deny-Non US [webmasterworld.com]
Banning NON NA IP's [webmasterworld.com]
Blocking Several Countries [webmasterworld.com]
Block Country [webmasterworld.com]

The above are just examples of primary beginnings and ALL require a more comprehensive effort.

It should be note that most people find the concept of grasping and editing Rewrite lines of IP ranges quite mind-boggling. WHERE even a solitary syntax error will result in taking your server down with a 500 error.

Some syntax errors don't result in a 500 error for months and then one day you make a simple adjustment (correctly) and a 500 takes your server down.
If your htaccess is few lines, this 500-server-down does not present any real issues, however when your htaccess is thousands of lines, review could take days and result in distorted vision and migraine headaches.

In the end, each webmaster must determine what is beneficial or detrimental to goals of their individual site (s).

[Sally Stitts]

Wow. Thank you.
What a definitive post, plus great links.
I will be busy studying.

[HuskyPup]

Prevent AdSense ads from being served, thereby improving all my AdSense stats

It's possible you may improve CTR and eCPM however it is also highly probable that overall earnings may be reduced since you will not be accruing ANY of the lower value clicks.

Don't make the mistake of assuming that all you will get are higher value clicks, SmartPricing doesn't work like that.

[ergophobe]

And of course, make sure you block caching in your robots.txt or meta tags, so that you don't have a cached copy out there that they can access no matter what you do on your server.

[jomaxx]

It may reduce content theft, but like HuskyPup I am skeptical it will "improve" your AdSense stats. It may also cost you link pop if you start banning countries where you've achieved popularity.

Besides, if I were Google, I would penalize sites that do this. They serve the entire world, not just the USA, and it's a major annoyance for their users when the SERP listings are inaccurate or unusable.

[TheMadScientist]

However,you will be anyway showing in Google, and if they want to copy your site they can do anyway by using a proxy from a country you do not block.

And they won't have the original source to compete with, so it'll be way easier for them to rank for the terms you are popular for, which IMO is added incentive to copy and scrape...

I am quite popular in India and Pakistan, and I don't want to be. I feel that showing my pages over there is costing me money, additional competition and lowered ranking.

1.) Can I ask how it's costing you money? (Bandwidth or something?)

2.) WRT additional competition, IMO if you don't want competition you're in the wrong industry... This is one of the most competitive environments I've ever competed in, because it's 'the wild wild west' where 'no blood, no foul' is the only rule, and it's rarely well enforced.

3.) How are your ranking lowered, or why do you perceive them as lowered?

The questions are real questions I have, because I really don't understand how serving a site in a country where it's popular could diminish it's rankings and IMO just about any ad clicks should cover hosting, so I really don't understand where you're coming from and would like to...

I don't use AdSense and have thought about blocking countries from accessing sites, but decided not to, so it's interesting to hear how you are drawing your conclusions on things. Is it purely bandwidth and AdSense or are there other factors you are looking at I'm missing?

[blend27]

Besides, if I were Google, I would penalize sites that do this.

jomaxx, I truly appreciate your opinion on country based blocking, but it would not make a financial sense for Google to penalize sites that do this. If the "user" clicks on SERP Link and leaves the GORG SERP, there is less monies to be made from Ads.

For some reason I believe that Sally's list(or some sort of it) of the countries to allow access from to the site is a very common practice among SMB Ecommerce stores. I don't think G should use that in the formula to rank the sites based on that fact. I could also understand the user frustration.

I've blocked India(and several other countries) close to/over 5 years ago on our main store that sells certain widgets that are very popular among women population. I posted a note with the email address to contact us directly with request to allow them access but also to specify a reason for the request.

To this day I have unblocked 2 IP ranges for the people who actually expressed interest in widgets. And at the same time ignored over 600(referrers are from page 34 in SERP for "link exchange +widgets + add url") of them that contained a reason as to them evaluating our site for a possible link exchange, guaranty top 10 in Google SERP, PPC Management (just last monththey've noticed that our copy on the ads was not good enough - this site used AdWords for 2 month 6 years ago) and some occasional promotions and timely business opportunities(I actually waited till promotion time limit expired and only then ignored it ;)) … See what I mean.

All though the original blocking decision was based on the Extra Abusive Bot Activity, constant Comment/GuestBook Spamming and several attemps to place orders with STOLEN CCs. Again this was not just for India.

[jomaxx]

I wouldn't be opposed to blocking by referring page, depending on the page, and at the minimum any e-tail company should have IP tracking to know where their orders are originating from geographically. It just seems like a blanket countrywide ban is overkill for the reasons I stated.

I very much doubt Google do any algorithmic penalizing of sites that do this, but don't forget they have mechanisms for manual feedback on the search results quality as well. Who knows what they might do if someone complained? Google even have employees in some of these countries (India, possibly still in China). I can't imagine they're too happy when sites block them on the assumption that they're thieves or pirates.

[wilderness]

I can't imagine they're too happy when sites block them on the assumption that they're thieves or pirates.

Neither Google (or any other SE), or visitors are required to be provided with an explanation for denial of access. More often than not, the visitors browser will simply return the browsers own default 403.

I limited non-NA traffic to two websites for more than ten years, including on the translators, SE provider tools and many other file utilities.
And all of this visitor restriction/denial without hindering my SERP's or even slowing down the crawling by the major SE's.

In all fairness, and frankly to be honest, I've never used Google Adsense or any other type of google ads on my sites. As a result, I'm without any capability to provide insights on how this access restrictions would effect these programs.

Additionally, the hogwash about the WWW being a community of free and unrestricted access is pure hogwash.
The WWW is served with a variety of uses and even closed networks with no public access at all.
"Extranet" and "Intranet" are good reading topics for those who may not be aware of such things.

[jomaxx]

Neither Google (or any other SE), or visitors are required to be provided with an explanation for denial of access. More often than not, the visitors browser will simply return the browsers own default 403.

The 403 is a pretty strong hint that your IP address is banned. I'm sure that Google employees have access to tools that would allow them to quickly confirm what is going on.

I limited non-NA traffic to two websites for more than ten years, including on the translators, SE provider tools and many other file utilities.
And all of this visitor restriction/denial without hindering my SERP's or even slowing down the crawling by the major SE's.

Just because traffic hasn't gone down doesn't mean that your actions haven't hindered the growth of your site.

Additionally, the hogwash about the WWW being a community of free and unrestricted access is pure hogwash. The WWW is served with a variety of uses and even closed networks with no public access at all.
"Extranet" and "Intranet" are good reading topics for those who may not be aware of such things.

Nobody said anything of the kind in this thread. We're talking about the overall user experience of geo-crippled websites and whether they should be listed in Google. They obviously should NOT be listed in regional Googles where users are banned, but unless they start spidering from every country in the world they have no way of being that specific.

[tangor]

There's a problem with egalitarian desires for the web. Not all fit into that mode, and G, B, and Y, and what's left of all the others, recognize that. The so-called "user experience" is equally limited, if by nothing more than language barriers.

Long and short, providers are fully justified to limit access ("no shirt, no shoes, no service"). And we do this all the time as regards bots, rippers, spammers, etc. Continuing that scenario there's nothing wrong in denying countries, ip ranges, or even little Johnny Snot-Nose.

There's no entitlement to access websites, though Google (in particular) seems to suggest that's the way it should be. I built my house (website) and I have every right to decide who enters. Period.

[wilderness]

The 403 is a pretty strong hint that your IP address is banned. I'm sure that Google employees have access to tools that would allow them to quickly confirm what is going on.

403's may be served based upon a variety of conditions, even multiple conditions. For google or anybody else to assume that all denials of access are based upon IP's is a bit far-fetched.

Additionally google doesn't have a crystal ball which gives them a capability to read other servers configurations. Nor do google employee's have an Ouija board.

[Rosalind]

If you want to prevent scraping, ban both the search engine cache and archive.org.

<rant>
These hundreds of lines of .htaccess for geolocation is a real bugbear of mine. How much easier would it be if IP addresses started with a country code, like phone numbers do? Webmasters should lobby to have this included in the spec for the next version.
</rant>

[jomaxx]

403's may be served based upon a variety of conditions, even multiple conditions. For google or anybody else to assume that all denials of access are based upon IP's is a bit far-fetched.

You're addicted to straw man arguments -- I said nothing of the kind. I was talking about a human being getting this error, not a brain-dead spider. If I was in India or China and this happened (and I was halfway interested in the site), I'd immediately try accessing the site through proxies. It would only take a few seconds to determine whether the site is selectively allowing access based on IP address.

[wilderness]

We've veered dramatically from the GEO topic and I apologize, however your submissions could easily be interpreted to both advise and read that GEO and/or denial is ALWAYS detrimental to websites, and it's simply not so.

FWIW, most proxy servers utilize malformed User Agents and are easily defeated utilizing simple white-listing procedures.
I've (as do many other webmasters) extensive lists of proxy server, server farms and even colo's, most of which are long denied.

How you or any other visitor (human or otherwise), could make an authoritative determination confirming that they'd been denied based solely up on IP is beyond my comprehension.

In any event our mis-communication is not beneficial to this thread and I'll stop here.

[outland88]

Your simplest solution might be in developing an excellent DMCA form letter you can fax Google. With a good form you can just plug the addresses in and be done in no time. Most of the time Google will knock infringing sites out in less than 48 hours. Also many leading hosting companies have IP filtering. If you’re selling physical items fraud screening is utilized by the leading processors.

[Silvery]

Instead of blocking your site from displaying to people in those countries, you might just write code to opt not to display the ads to IPs in those areas.

Or, opt to display ads from some other network to people in those areas. Perhaps some Asian/Indian ad network?

[SEOPTI]

I would not ban archive.org as suggested. I remember MC saying most spammers ban archive.org

[KenB]

I've had archive.org banned for maybe ten years or longer. Most spammers may ban them, but so do a LOT of legitimate sites. They are no different than scrapper sites as far as I'm concerned. They violate the robots.txt file and if I did allow them in they'd provide no benefit to me in exchange for consuming vast quantities of my server resources.

[incrediBILL]

There are several free sites available such as, [blockacountry.com...] , where you click a range of countries and it generates a nice IP range output to drop in your .htaccess file, no fuss, no muss, no programming.

[KenB]

One thing to remember about banning countries based on IP addresses is that IP address ranges do get reassigned from time to time. As such any static solution like a .htaccess block list will become outdated over time. In the short run it takes a little work to set up, but in the long run the easiest to maintain method of blocking would be using a geo IP database/CSV file to convert IP address to country code via PHP (if you are using PHP) and then block/allow based on the resolved country code. Then when it comes time to update your IP address data all that is required is to drop the new data into place.

[pbaddock]

If you are worried about your htaccess being slow, or you want to block more than port80/http traffic, then you could always add the IP's to your firewall (iptables) .. this has more potential than just blocking http/port80 access, in that if you are repeatedly getting hack attempts on say ftp, telnet, ping or other, from China, Russia or anywhere else you can adequately block them.

For a recent DDOS I faced I had to do this. There are several ways to implement GEO IP lookup, script which inserts/modifies the firewall in a similar manner as to you adding to the .htaccess

for port 80, the benefit is that the traffic never even gets to start a http/apache process = no speed decrease.

[IanCP]

Prevent AdSense ads from being served, thereby improving all my AdSense stats

Sally, as HP observed earlier it may not improve AdSense in any practical way. Possibly some variations in CTR and eCPM but surely the bottom line is what really counts.

Do you seriously expect that to improve by blocking certain countries?

As for copying, well there are ways any determined copier will find around your efforts. I sympathise, but it's a fact of life we mostly have to live with.

Copy theft, translated into another language is nigh impossible to detect and if detected, impossible to have taken down. Try Russian ISP's for starters.