homepage Welcome to WebmasterWorld Guest from 54.167.174.90
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

This 37 message thread spans 2 pages: 37 ( [1] 2 > >     
Ban 75% of the Planet
pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 4:16 pm on Jun 23, 2008 (gmt 0)

Let's say that I'm a local business here in California. And, that 98% of my product is shipped within the United States and ordered by consumers residing in the United States. What benefit is there for me to allow anyone other than those within a certain IP range access to the site? I don't want visitors from the other side of the planet. I really don't want them if they our outside our serviced areas really. We don't ship product there, never have and probably never will. So, I'm going to block those countries from accessing regionally specific sites.

Am I making a rash decision? Talk me out of it. We're already starting development on the 403 process right now. In fact, we have the logic in place and my programmer warned me to test this slowly. :)

In my thirteen (13) years of doing this, I never, ever thought I'd be starting a topic like this, ever!

Our new Website Whitelist. Actually an IP Whitelist, that sounds better and was available!

Everyone is out there focusing on their Blacklist, why not look at this from the Whitelist perspective? Am I missing something other than a few visitors here and there?

 

Staffa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3681594 posted 4:54 pm on Jun 23, 2008 (gmt 0)

I'm not the one who will talk you out of it, I've been doing it for a number of years.

Your reasoning is sound and logical so why not do it ?

I block for a different reason, some to do with sales and I have whole countries and part continents blocked. OK, I don't have 'thousands' of visitors but neither do I have thousands of problems. ;o)

trillianjedi

WebmasterWorld Senior Member trillianjedi us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 5:12 pm on Jun 23, 2008 (gmt 0)

Am I making a rash decision?

I see no reason whatsoever not to block everyone bar a single country if visitors outside of that Country bring you no value. I don't have a single geographic website. But, if I did, that would be exactly my approach...

pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 5:40 pm on Jun 23, 2008 (gmt 0)

I'm not the one who will talk you out of it, I've been doing it for a number of years.
Your reasoning is sound and logical so why not do it?

Okay, that was a tug in the right direction, do it, do it...

OK, I don't have 'thousands' of visitors but neither do I have thousands of problems.

I think I can trim out a thousand or two that really don't need to be there. The site may not serve them any purpose. And, because of the amount of abuse coming from that particular area, its not worth it. I'm going to do what any good SE would do, ban the buggers!

I see no reason whatsoever not to block everyone bar a single country if visitors outside of that Country bring you no value. I don't have a single geographic website. But, if I did, that would be exactly my approach.

Okay, I think I'm convinced now. :)

I see no reason NOT to do it either. In fact, each character I type here solidifies my decision to do so. I think in the long run, everyone will win, including the Search Engines that we rely on for our valued visitors.

And, how many sites are really that "global" in nature? I mean, there are probably a good percentage of us reading right now where this may be a valid idea moving forward. We've thought about it from the Blacklist side of things, let's look at it from a Whitelist perspective. Same thing I guess...

Receptional Andy



 
Msg#: 3681594 posted 5:47 pm on Jun 23, 2008 (gmt 0)

I think this thread needs a counter-argument ;)

The reason not to do it is false positives. Whether the certainty of blocking a (likely small) amount of genuine visitors/customers is worth the advantage of blocking undesired visitors is for a individual site to decide. IMO the potential risk is both in lost sales and damage to reputation.

I'm often surprised to see genuine sales on UK-only sites from overseas visitors (mostly US, but a European flavour in some cases too).

rocknbil

WebmasterWorld Senior Member rocknbil us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 5:50 pm on Jun 23, 2008 (gmt 0)

The only reason I can think of is the possibility legitimate sites from those areas may bring links and business to your site. They may not purchase from you, but they may link to you. (Weak point at best.)

Or, California customers traveling in those countries. I hear it happens. :-)

We have a long list of banned IP's, don't think it amounts to 75% though. We just give them the boot as they come along, and are still minimalist in banning ranges.

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3681594 posted 5:59 pm on Jun 23, 2008 (gmt 0)

Allow me to play Devil's Advocate for a moment.

It's a small planet and getting smaller. People fly around it in vast numbers every day. Many of those do so on business and carry their laptops with them. They are by definition high spenders. A lot of them spend time in hotel rooms abroad and might like to shop in their spare time. Then there are your boys in uniform stationed around the globe, all of whom also call USA home.

OK I'm done. I would probably ban most of the planet too if I were in your shoes.

how many sites are really that "global" in nature?

All of mine - that's what attracted me to the medium in the first place.

...

pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 6:17 pm on Jun 23, 2008 (gmt 0)

The more and more I get into the reviewing of 404s and such, I am now convinced that it is the right thing to do for a handful of my clients. The amount of abuse far exceeds any value a visitor from that particular region may bring for some clients. And, if the client is truly that global in nature, I'm going to recommend country specific sections of the site only served to that IP range. I know, I know, false positives, proxies, etc. I'm sure I'll learn as I go and I'm not going to just flip the switch without getting valued feedback from my peers here at WebmasterWorld.

There are at least five countries, two of them rather large, that are getting added to the Blacklist though, I know that for sure. We're through with you sucking up bandwidth, copying content, stealing designs, generating 404s and 500s with your freakin' probing, etc. I don't want to have to deal with this at the micro level. Nope, if the abuse from your country exceeds my comfort level, c'ya!

And then we'll get into the local abuse after that. Deal with the large stuff first and focus on the bad boys at home!

403 Forbidden

Repeat to self... "Edward, you are being totally irrational, you know that don't you?"

oddsod

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3681594 posted 6:30 pm on Jun 23, 2008 (gmt 0)

Is Google tracking clicks? Will non-US traffic get a 404? How do you see this impacting on your SEO?

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3681594 posted 1:04 am on Jul 3, 2008 (gmt 0)

My primary focus at Webmaster World for some years has been the SSID Forum.
When Jim began the Apache Forum, I followed their as well, however my submissions there have been few.

Bill requested that I pop in here.

pageoneresults,
There's really nothing wrong with your thought.
Major Corporattions and many paid access sites been doing so since the beginning of network access. The only difference between major corps (and their paid Network technicians) and us?
Seems to be our vanity ;)

Do a google on "extranet"!
Please look beyond the Wiki page which provides a sufficient explanation for most.

How do you see this impacting on your SEO?

oddsod,
I've no clue what google tracks, however imposed Extranet focuse for some years hasn't hindered my SE listings (even whilst eliminating major non-North American SE bots from access).

Another thought for focus could narrow down visitor referrals based on "topic". And localised topic.

Don

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3681594 posted 1:23 am on Jul 3, 2008 (gmt 0)

Allow me to play Devil's Advocate for a moment.

It's a small planet and getting smaller. People fly around it in vast numbers every day. Many of those do so on business and carry their laptops with them. They are by definition high spenders. A lot of them spend time in hotel rooms abroad and might like to shop in their spare time. Then there are your boys in uniform stationed around the globe, all of whom also call USA home.

OK I'm done. I would probably ban most of the planet too if I were in your shoes.

how many sites are really that "global" in nature?

All of mine - that's what attracted me to the medium in the first place.

Samizdata,
Although unresricted access of the entire internet is dreamer's theory and by nature, a pure thought?

There restrictions across most every IP range and/or website that we come across?

Under the "unrestricted access theme", one and all would have open access to major goverment and/or nations sceurity websites?
Most everybody relaizes that isn't about to happen and as a result, what is the difference between the above imposed access restrcitions and any that "we" may determine as beneficial to the focus or of our own websites?

pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 1:37 am on Jul 3, 2008 (gmt 0)

wilderness, thank you very much for dropping in, much appreciated! I always like it when someone sides with me. ;)

How do you see this impacting on your SEO?

In theory, I would expect it to have a positive impact. I "know" those scraper sites are having a negative impact in the overall scheme of things. I know those cloaked 301 redirects are having a negative impact. All that freakin' crap that comes from specific locales needs to stop and I'm going to do it. If I block a few "valid" visitors in the process, I don't care anymore. Yes, it is that serious. For me to say "I don't care about the visitor" is unheard of. But in this case, its more of a parasite than anything else.

Yes! I called you out. You, the one lurking, reading this topic. The one that has stepped up 404 production for one of my properties just to mess with me. That's okay, I've got something for ya once we figure this all out. I'm going to let you see yourself in the mirror. ;)

There was a post today at Search Engine Roundtable where the headline reads...

Google Says Blocking Countries Outside of the US is Against Policies
2008-07-02 - [seroundtable.com...]

A quote from John Mu and then Barry...

Googler, JohnMu, stepped in to say that by blocking an entire region, it would "be considered cloaking" and would be against Google's Webmaster Guidelines.

Got that, if you block specific regions of traffic, like everyone outside of the US, that is cloaking and against Google's guidelines.

I wonder if John Mu read this topic. Why would they be so vocal about that? What? Is there some concern that I'm cutting off revenue stream by doing this? I totally believe that all these parasites from outside my targeted regions are making more money for Google than my clients are making with their websites. I really do. ;)

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3681594 posted 2:19 am on Jul 3, 2008 (gmt 0)

unresricted access of the entire internet is dreamer's theory

I entirely agree Don, and I certainly don't offer unrestricted access to my sites.

I do offer global access because my target audience is worldwide, but I respect the right of any webmaster to allow or deny access as they see fit - they pay the piper, they call the tune.

It seems perfectly reasonable to me for pageoneresults or anyone else to disallow traffic from IP ranges that send them nothing but trouble - I do some of that myself (and learned how from the kind folks of WebmasterWorld such as yourself).

I don't think Google has any business telling us which IPs we can and cannot block.

it is certainly not "cloaking" in any accepted sense of the word.

...

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3681594 posted 3:08 am on Jul 3, 2008 (gmt 0)

Googler, JohnMu, stepped in to say that by blocking an entire region, it would "be considered cloaking" and would be against Google's Webmaster Guidelines. Got that, if you block specific regions of traffic, like everyone outside of the US, that is cloaking and against Google's guidelines.

See, another fine example of Google trying to tell people how to do business, it's a BIG BROTHER mentality all over again.

Instead of being a big bully, why doesn't Google provide a facility to allow us to specify what country our sites should display in, just like they do with AdWords.

What naive Googler JohnMu doesn't get is many of the problem countries use dynamic IP pools and proxies so trying to profile the behavior of someone doing something bad doesn't work when every single access is jumping to a new IP which leaves you no choice but to block the entire service they use since nothing else will help.

FWIW, I'll continue to block entire countries that abuse my site and are the main source of sending spam and perhaps Google should keep out of how other people run their business and focus on their own backyard.

Sheesh, the nerve of some monopolistic conglomerates, it's enough to make you scream.

[edited by: incrediBILL at 3:09 am (utc) on July 3, 2008]

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3681594 posted 3:13 am on Jul 3, 2008 (gmt 0)

Not to mention, from an ecommerce point of view, if you block all the countries you don't ship to you eliminate a bunch of whiny emails begging you to make an exception and you also block the most frequent method of fraud which is using a foreign IP address so it can't be traced with a simple subpoena.

Like I said before, it's Google's fault that they don't allow you to restrict which countries your results are displayed in, not mine.

The irony is in every country except the US they offer local only search such as:
Search: * the web * pages from Canada

If you only want to search the US it's TOO BAD because Google.us redirects to Google.COM!

HELLO? ARE YOU PEOPLE PAYING ATTENTION AT THE 'PLEX?

Wacky.

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3681594 posted 3:38 am on Jul 3, 2008 (gmt 0)

Google should keep out of how other people run their business

Are we being lectured by a company that restricts web access by the people of China?

Or am I missing something here?

...

pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 3:52 am on Jul 3, 2008 (gmt 0)

What naive Googler JohnMu doesn't get is many of the problem countries use dynamic IP pools and proxies so trying to profile the behavior of someone doing something bad doesn't work when every single access is jumping to a new IP which leaves you no choice but to block the entire service they use since nothing else will help.

Dude, arrrggghhh, chasing those damn probing peckerwoods is like delving into a black hole. I'd really like to provide a list of countries in this topic but that would upset quite a few and I'd be violating TOS. I will tell you that there are US based servers that are going to get banned in the process. Anything originating from those IP Blocks is on my list of to-dos. I'm coming to get you so now is the time to take your best punch damnit! :)

I am so happy to see you guys here supporting me, I really am, thanks! This whole lawlessness of the Internet has really gotten to me lately and I plan to do something about it, at least for those websites that I have control over. You can keep all that bullschit in your own ccTLD and leave me out of it.

After I'm done with all this, I want to provide a real time "heat map" of the offending locations for all to see. Yup, you pissed me off that much. Time for the tide to turn. I remember Brett talking about this years ago, heck, back in the 90s. I wish I knew then what I know now. Its always been in the back of my mind though. Now it is at the forefront, watch out...

Some choice naming conventions... Leeches, Bottom Feeders, Parasites, Disease, Fungae, Barnacles, anything that latches on and sucks the life out of life.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3681594 posted 5:03 am on Jul 3, 2008 (gmt 0)

I want to provide a real time "heat map" of the offending locations for all to see.

Cool idea, I should've though of that before.

Using Google Maps just to drive the point home to them as well... hehehe

Ocean10000

WebmasterWorld Administrator 10+ Year Member



 
Msg#: 3681594 posted 5:13 am on Jul 3, 2008 (gmt 0)

I wonder how much data that map can handle before it buckles? I have data going back several years. I know I won't have time to test it this weekend.

pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 5:15 am on Jul 3, 2008 (gmt 0)

Aw man, I spilled the beans, my programmer is going to hang me! ;) Hold on, I've got a zinger for ya...

Oh, I don't want to have to dig all that up right now either. But, I have a multitude of topics that all tie into this one topic. There is the Tracking of 400 and 500 Errors which I've been talking to myself in. And then there are multiple topics where I mention the term "competitive sabotage". Googling both will get you what I'm bringing together. Its a tedious process too... But well worth every minute spent!

And now I see the whole social networking scene ablaze with similar topics. Hey, where ya been? And start giving some damn credit to the original source would ya? ;)

I wonder how much data that map can handle before it buckles?

Is that possible? ;)

I'm sure you'll be adding all the proxy servers to the maps?

Staffa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3681594 posted 7:21 am on Jul 3, 2008 (gmt 0)

incrediBILL, you just saved me a lot of typing by putting down my thougts exactly.

What countries and IP ranges we block on our sites is NONE of Google's business. They should also remember that they need all of our sites to run a SE. Should the day come that G gets blocked THEY will be out of business.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3681594 posted 11:54 am on Jul 3, 2008 (gmt 0)

They should also remember that they need all of our sites to run a SE. Should the day come that G gets blocked THEY will be out of business.

Webamsters utilizing visitor restrictions, whether spammers, harvesters or unwanted bots (even regions) are a very, very small count in the overall website picture.
We here, tend to exaggerate our presence due to the conversation of a collective group.

Staffa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3681594 posted 12:27 pm on Jul 3, 2008 (gmt 0)

Wilderness I agree with you but that doesn't change it that it's none of G's business who and what we block. G just stands for Google and not G*d.

vincevincevince

WebmasterWorld Senior Member vincevincevince us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 12:38 pm on Jul 3, 2008 (gmt 0)

My case against this is financial.

If you sell widgets in California, but get lots of traffic from other people looking for widgets...

...redirect them to an affiliate programme which does serve them.

Why throw away valuable traffic? Use your geolocation and make money out of these otherwise dead leads.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3681594 posted 12:55 pm on Jul 3, 2008 (gmt 0)

vince,
There's NOT any "rule of thumb" here.
Each webmaster must determine what is beneficial or detrimental to their own site (s).

it's none of G's business who and what we block

Staffa,
I agree.

Further, unless google makes an extreme attempt at cloaking themselves, how would they determine that specific regions were denied? (i. e., it's OK for Google (and other bots) to cloak; "per say", however not webmasters).

We should also assume that it's OK for Google to allow a dual standard, when they potentially failed to penalize some sites for not allowing full and open access too all corners, while penalizing others?
Hardly!

pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 2:09 pm on Jul 3, 2008 (gmt 0)

We here, tend to exaggerate our presence due to the conversation of a collective group.

Some of us within the group tend to exaggerate more than others. Moi? ;)

I go to sleep thinking about this and I wake up thinking about it. It has engulfed me. Some might say I need to step away from it and take a break. I don't think so. Once I latch onto something I want to see it through to the end if I can, get er done!

So, I decide to check some 400 statistics just now. From 20080701 at 0001 to 20080703 at 0700 there were 1,023 404s generated for one site. Of those, I'd say more than 75% of them are from offshore IPs that are probing for various destinations. Some of them I look at and wonder, "what the heck are you looking for?" Others are easy to determine. For example, anything with /javascript is a sign that someone/something may be probing. Get out of here! What are you probing for? Looking for vulnerabilities? You piece of ****!

I mentioned earlier that the lurkers were watching me. Well, it is rather coincidental that 404 production on a couple of my "visible" websites is up quite a bit over the past 30-60 days, 454 in the last 12 hours for just one site. I do firmly believe that some of the culprits are right here at WebmasterWorld and a few other places I trek. They lurk day in and day out. They learn much from this community. They also take advantage of it. Its a given, there is really no need to even debate it. ;)

Back to the "Heat Maps". I'm serious about our project, are you?

pageoneresults

WebmasterWorld Senior Member pageoneresults us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 6:49 pm on Jul 3, 2008 (gmt 0)

Just a quick update from Google today on this...

Google Retraction: Blocking Regions Is Not Cloaking
2008-07-03 - [seroundtable.com...]

From John Mu in a Google Groups thread...

After a bit of double-checking, I have a clarification where I was mistaken. Sorry about the confusion! The important part is that you do not treat the Googlebots any different than other users from that region. So if your site blocks users in the region where the Googlebot comes from (based on the IP address and your IP/Location lookups), you should be blocking it as well. Blocking users outside of the Googlebot's region would generally be ok.

Okay, now I don't have to worry about the hand so much. ;)

LifeinAsia

WebmasterWorld Administrator lifeinasia us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3681594 posted 10:39 pm on Aug 6, 2008 (gmt 0)

What benefit is there for me to allow anyone other than those within a certain IP range access to the site?

Just to play Devil's Advocate: What if I buy your product and try to use it on vacation (or a business trip) while outside the U.S.? (Or I am in the military stationed overseas. Or I am a foreign citizen and bought it while I was living/travelling in the U.S.) I'd be mightily peeved if I couldn't access your site to get some support.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3681594 posted 3:09 pm on Aug 7, 2008 (gmt 0)

Just to play Devil's Advocate:

What if I pop in to an APNIC/Asian website that has linked and/or copied from my ARIN/North American website and all I see are charcaters in a language I neither have the capability to read or translate?
And Vice Versa for APNIC/Asian folks visiting ARIN/North American website?

The internet is a highly focused multi-netwok community today of open and closed traffic, open and closed regions and open and closed sites.

There are no "generalities" apllied to every instance or website.

Each webmaster must determine what is beneficial or detrimental to their own site (s).

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3681594 posted 3:16 pm on Aug 7, 2008 (gmt 0)

p1r, I think you deserve the distinction of being nominated as the the biggest $h!T disturber on Planet Earth.

Counterpoint: What if any of those IPs you're blocking would have given you some PR6 links to your site if they had been able to access it?

That said, unless there's good reason to think there's anything to gain, on a high traffic site, why spend $ on tons of wasted bandwidth? Unless there's some compelling financial advantage, it just seems to make good sense to block access outside of a reasonable targeted area, as a sensible business decision.

[edited by: Marcia at 3:22 pm (utc) on Aug. 7, 2008]

This 37 message thread spans 2 pages: 37 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved