Decided to block Googlebot after Sep/12th traffic plunge

[acutrician]

Has anyone else experienced a sharp traffic reduction (in Canada) from the 12th to the 13th of Sep/2014? My 14 year-old site has been hit by several Panda updates since Feb/2011, but so were lots of other sites. However as much as I have checked around, Sep/12-13th doesn't seem to coincide with any known, or named updates that I could find. Nothing has changed on my site, and there were no warnings or messages in GWT.

Google visits plunged by 60-70% overnight, while Bing and Yahoo stayed the same, so my traffic is down about 95% since Feb/2011.
Since I don't rely on Google for a living (it's a research/info site), I decided to return the favor and block Googlebot from spidering my site now. I'm hoping to get out of the Google index to a point where my site (11,400 hits right now) will not come up any longer through a Google search, which is probably not entirely possible.

Again, just curious if the Sep/12th traffic plunge was experienced by anyone else (similar type of site?), or if it was due to a random algo change.

[Clay_More]

O.K., Google is the best S.E. ever and can immediately discern quality.

So why would someone seek footer links that were broken into "going to" "Europe"?

Isn't that obvious manipulation that should be immediately disregarded by a quality S.E.?
I have no issue if someone is whitehat or some other hat, but don't preach one thing and do another.

[JD_Toims]

Isn't that obvious manipulation that should be immediately disregarded by a quality S.E.?

Well, going with Google's "penalty mentality", the perpetrator should be penalized by the best, biggest, most-influential SE, rather than simply having their shady efforts to rank better being discounted, but of course "penalizing" a "vocal believer" who says one thing while doing another might have some unwanted ramifications, so it's probably "best business" for Google to leave things alone and pretend it's algo is spot-on and not being gamed by one or more of it's biggest supporters.

---

I mean, it's not like if the average of 1+ updates a day in 2013 really improved the results Google's market share based on overall queries should drop, because people found what they're looking for with less searching or anything like that, really, Google's market-share stays the same even after all the updates to improve quality and not get gamed, because it's the best and people find what they're looking for with the fewest queries of any SE...

---

M$ didn't emulate Apple's Mac OS with Windows, because M$ had the best OS on the market, M$ did it, because regardless of market-share, M$ was completely and totally beat by Apple's Mac OS -- Market-share is *not* always indicative of quality, if it was, a clunky DOS based OS would be the standard "everyone" looked for in their next computer, but for some reason it's not. Go figure...

[edited by: JD_Toims at 11:51 am (utc) on Oct 12, 2014]

[iammeiamfree]

One of the best ways to stick it to google is to offer a selection of quality links at the bottom of your pages with more in a directory. What google have done with nofollow and getting webmasters into a state of fear about linking out with their disavow emails and so on is basically monopolistic behaviour like Monsanto engineering GMOs so only their crops will grow in a treated field. The best way to respond is to have every website offer quality outbound links relating to their niche and to turn the web into well a web rather than a controlled and limited media empire. At some point visitors are going to leave a site and it is better they go to a site recommended by webmasters than back to search engines. Search engines are less important when every site takes on a part of that role. Instead of 90% google it should be 100% webmasters. If you go to google and click the first result you will find listings on the site to other useful sites on that site without needing to go back to google. All roads lead to Rome and I am the emperor so add a link today so people can get where they want to go. It is like in Venice they put the sign to Rome even if they don't want people to go there.

[EditorialGuy]

So why would someone seek footer links that were broken into "going to" "Europe"?

Does anyone seek footer links these days, outside of bottom-feeding SEO firms whose Bible is SEO for Dummies: 2003 Edition?

And what does that have to do with blocking Googlebot (and turning your back on the world's largest search audience in the process)?

So why would someone seek footer links that were broken into "going to" "Europe"?

Does anyone seek footer links these days, outside of bottom-feeding SEO firms whose Bible is SEO for Dummies: 2003 Edition?

And what does that have to do with blocking Googlebot (and turning your back on the world's largest search audience in the process)?

I wonder how many people here who encourage others to block Googlebot, cut the Google cord, etc. have actually done it themselves?

[JD_Toims]

Does anyone seek footer links these days, outside of bottom-feeding SEO firms whose Bible is SEO for Dummies: 2003 Edition?

So, hypothetically of course, if those links somehow happened to point to your site, we should take your word for it that you didn't build them, believe you're not a "bottom feeder", had nothing to do with them, and they are somehow simply a "gift" a competitor gave you for some reason?

If we take that position with you, shouldn't we [and the "almighty, most used search engine with the world's largest audience"] take the same benefit-of-the-doubt giving position with others to be fair?

And what does that have to do with blocking Googlebot (and turning your back on the world's largest search audience in the process)?

At the heart/principal of things, quite a bit, but somehow I'm sure some people will miss or pretend like they don't get what I'm saying, unfortunately.

[edited by: JD_Toims at 2:38 pm (utc) on Oct 12, 2014]

[EditorialGuy]

So, hypothetically of course, if those links somehow happened to point to your site, we should take your word for it that you didn't build them, believe you're not a "bottom feeder", had nothing to do with them, and they are somehow simply a "gift" a competitor gave you out of the kindness of their heart?

Yes, I've seen that happen. Some people use footer links as a blogroll.

I've never asked for footer links myself, so I can't say that I've spent much time thinking about them, but I'd guess that Google doesn't give them much (if any) weight.

Mind you, none of this has anything to do with blocking Googlebot. And unlike some here, I'm not questioning anyone's motives. Maybe the OP had an epiphany after his "Sept 12/traffic plunge." It's always easier to reject what you don't have than to reject what you do.

[JD_Toims]

Mind you, none of this has anything to do with blocking Googlebot.

Sure it does, but there's that whole "forest/trees thing", so people here missing it the same way Google's engineers do isn't very surprising.

[iammeiamfree]

Blocking googlebot is no big deal if you think you can rely on some other traffic source or revert to some job in a fast food restaurant if necessary. If you've got real balls then try travelling with no money for an extended period or better still getting into permaculture and planting up a survival forest. Waiting for the rains is much more exciting that google updates and what the land can give much more rewarding I would say too. Anyhow for me atleast the sooner I could care less about the web the better.

[edited by: iammeiamfree at 3:24 pm (utc) on Oct 12, 2014]

[acutrician]

I wonder how many people here who encourage others to block Googlebot, cut the Google cord, etc. have actually done it themselves?

If you are hinting that I may be guilty of such practices -- nope.

Other than a handful reciprocal link exchanges in the early stages of building my website 14 years ago (which was common and not frowned upon by search engines at the time) I haven't bothered since. I don't know enough about the technical aspects of any of the darker SEO practices, and I have no interest in acquainting myself with them.

Right now my site still comes up when I search for my brand name, but with a note that "A description for this result is not available because of this site's robots.txt." Hopefully the "noindex" tag will kick in shortly.

I find all this quite fascinating and cannot wait to see how much it is possible to purge a website, and especially a brand name, from Google's SERPs.

[bhukkel]

Right now my site still comes up when I search for my brand name, but with a note that "A description for this result is not available because of this site's robots.txt." Hopefully the "noindex" tag will kick in shortly.

If you block your site with robots.txt, googlebot will never sees the noindex tag. It is better to allow googlebot to crawl the noindex tag . When every page is removed from the index disallow googlebot with robots.txt

[Samizdata]

cannot wait to see how much it is possible to purge a website, and especially a brand name, from Google's SERPs.

You can stop Google indexing your website very easily if that is what you want.

But removing your brand name from the SERPs is another matter entirely.

Other sites which mention it (or link to your site) are not under your control.

And they will almost certainly be indexed by Google whether you like it or not.

...

[netmeg]

I find all this quite fascinating and cannot wait to see how much it is possible to purge a website, and especially a brand name, from Google's SERPs.

It's pretty easy, except, as mentioned above, when other people mention your brand by name on their sites. I have had a lot of situations where clients' developers have accidentally (or ignorantly) allowed a dev site to be indexed while being worked on, and have had to go back in and remove all traces of it.

[acutrician]

If you block your site with robots.txt, googlebot will never sees the noindex tag. It is better to allow googlebot to crawl the noindex tag . When every page is removed from the index disallow googlebot with robots.txt

Glad you told me. Thanks!

I'm keeping JD Toims' 403/htaccess suggestion in mind as well.

[cabbie]

I blocked 2 of my popular sites completely from googlebot, when they were penalised for a unnatural link penalty.
It was my reasoning that Google's serps were poorer for not having my sites in it's index.They were info sites like acutrician's.
I was waiting for them to call and beg me to let them index my site.:)
alas, that never happened :( and after 6 months I removed the noindex tag.
They are now both back at number 1 for their subject matter.
Seems common sense (on both sides) won in the end.

[Clay_More]

Does anyone seek footer links these days, outside of bottom-feeding SEO firms whose Bible is SEO for Dummies: 2003 Edition?

Evidently some people do while portraying themselves as purer than snow white hats. Was that link because you hired a bottom-feeding SEO firm whose Bible is SEO for Dummies: 2003 Edition?

[EditorialGuy]

Clay_More, if you'd take off your SEO blinders, you might realize that, in the real world, people get unsolicited links in all kinds of ways. For example, a news site began linking to our site a number of years ago every time it ran an article on [topic] (We're listed as a related resource}, and our total number of links from that site was in the thousands when I last checked.

As for footer links, our site used to have a section of about 40 pages where every page had a footer link to an academic library and a government agency, because we were using public-domain resources from those sites and wanted to credit them.

If you think all footer links are spam, it's probably because you've used them for that purpose--or because you've spent too much time listening to people who do.

[JD_Toims]

if you'd take off your SEO blinders, you might realize that, in the real world, people get unsolicited links in all kinds of ways.

So, are you saying even though when people look at your link profile and can easily conclude it looks unnatural, you actually had nothing to do with it, meaning you should not have a penalty applied to your site, even though that's contrary to the "penalty mentality" exercised by the biggest, most influential SE on the planet?

If the preceding is correct and you should be given the benefit of the doubt rather than being penalized, why is it you support a SE which penalizes sites rather than giving them the same benefit of the doubt as you think you should be given as much as you do?

[mrengine]

I'm not questioning anyone's motives. Maybe the OP had an epiphany after his "Sept 12/traffic plunge."

Why do you say you are not questioning anyone's motives but yet say "maybe the OP had an epiphany?" Is some type of an awakening not a "motive?"

It seems that you like to create a lot of controversy around here with your statements - many statements that contradict themselves and/or are so far removed from reality that many are left questioning your motives.

It's the OP's website and he is free to do with it as he chooses just as you can with yours. If Google is not sending him much traffic, there's really nothing gained or lost by the OP blocking Google. In fact, I rather like the idea and can see how blocking Google gives some webmasters the ability to gain some control back over their destinies - by forcing them to rely less on Google's predictable profit-driven algorithm changes that tend to nickle and dime smaller website owners each and every time.

[EditorialGuy]

It's the OP's website and he is free to do with it as he chooses just as you can with yours.

So why egg him on, then?

If Google is not sending him much traffic, there's really nothing gained or lost by the OP blocking Google.

Maybe not today. But what about tomorrow? Or next week, or next month? Google's algorithm isn't set in stone, and if the OP's rankings have suffered because Google's algorithm is flawed (as most people here are quick to assume), then wouldn't it make sense to leave the door open for future improvements in the algorithm? Given how little it costs to let Googlebot crawl, so isn't that the more sensible and reasonable thing to do?

[JD_Toims]

Great question dodges, again.

...if the OP's rankings have suffered because Google's algorithm is flawed (as most people here are quick to assume)...

If it weren't your link profile would have you sitting on page 600+ with no traffic from Google like many people, which might change your tune -- It's amazing how many people sing the Google song while Google is working for them, even if it's due to a flaw on Google's part that keeps them ranking where they are. Talk about wearing blinders...

[Samizdata]

If Google is not sending him much traffic, there's really nothing gained or lost by the OP blocking Google.

There is certainly nothing gained - hence the initial response of "Don't see why you'd bother".

Conversely, there is certainly something lost - the 30-40% of Google traffic remaining after the "plunge".

The OP's stated intention is to get Google users to switch to other search engines.

But making the site unknowable to those very users seems a counter-productive strategy.

A more effective method might be to allow them access and attempt to persuade them directly.

Otherwise they will just continue to use Google, none the wiser.

...

[JD_Toims]

There is certainly nothing gained...

Not traffic gained from Google, but contrary to popular belief, traffic from Google, much like money to some, isn't everything.

Conversely, there is certainly something lost - the 30-40% of Google traffic remaining after the "plunge".

You keep jumping on percentage as if you know pre-drop traffic levels and know it must be a large number of visitors lost in an attempt to prove your point, but all we really know is the percentage and:

...so my traffic is down about 95% since Feb/2011

In a day and age where principal seems to be a lost cause for many, having some and standing for it can actually be as much or more of a benefit as compromising it so numbers sound better -- I know people who would visit, frequent, talk about and share a site, because it denied access to Google simply based on principal -- Would it outweigh the loss from blocking Google? Depends on the site...

[EditorialGuy]

If it weren't your link profile would have you sitting on page 600+ with no traffic from Google like many people, which might change your tune -- It's amazing how many people sing the Google song while Google is working for them, even if it's due to a flaw on Google's part that keeps them ranking where they are. Talk about wearing blinders...

You're grandstanding again, and you're ignoring my question:

wouldn't it make sense to leave the door open for future improvements in the algorithm? Given how little it costs to let Googlebot crawl, so isn't that the more sensible and reasonable thing to do?

As for standing on "principal" (sic), the OP apparently didn't mind traffic from Google until he started losing it, so "principal" doesn't really come into play here.

[acutrician]

and after 6 months I removed the noindex tag.
They are now both back at number 1 for their subject matter. Seems common sense (on both sides) won in the end.

That's a happy ending for some - while it lasts. For thousands of other websites, this hasn't, and will not happen.

Even top earners - who are smart - diversify because they know that the Google gravy train may derail any day, but the real smart ones make it their #1 goal to not let the ever-fickle Google traffic become the primary reason.

[samwest]

"When the elephants fight, it's the grass that gets trampled."

- Thai Proverb

...but it usually gets fertilized too!

[JD_Toims]

You're grandstanding again

Reality, grandstanding, call it what you will. My statement is still true.

...and you're ignoring my question:

You ignore over half the questions people ask you and many of the points people make here, so not only are you currently grandstanding, you're implying I should do something you don't bother with most of the time, which is hypocritical at best.

[Shepherd]

wouldn't it make sense to leave the door open for future improvements in the algorithm?

There's no one size fits all answer to this question. Every circumstance is different. On the surface, why pass up free traffic. But if we dig a little deeper and understand that traffic from google does not come without a cost many might come to a different conclusion. Allowing google access today may lead one down a road they may not be able to recover from in the future.

Our actions can affect google. Lets take a look at a niche, plane tickets. google moved into that space, their offering is mediocre. So what if, when google moved into that space, the big players already in the space removed their sites from the index and stopped buying adwords. How long would it be before searchers stopped searching on google for plane tickets?

[EditorialGuy]

Even top earners - who are smart - diversify because they know that the Google gravy train may derail any day, but the real smart ones make it their #1 goal to not let the ever-fickle Google traffic become the primary reason.

Diversification is good, but aren't you proposing to make your traffic sources less diverse by eliminating Google from the mix?

[Samizdata]

You keep jumping on percentage as if you know pre-drop traffic levels and know it must be a large number of visitors lost in an attempt to prove your point

I am merely quoting the OPs figures, which are all anyone has to go on.

I have no point to prove, my posts are intended to help a new WebmasterWorld member with the issue raised.

In that vein, I would also point out the logical fallacy in the idea that, if a site does not come up in Google SERPs in response to a query, users will switch and look for it on another search engine.

The fact is that Google users will not even know it exists, and will simply go to other sources of information.

...

[JD_Toims]

The fact is that Google users will not even know it exists, and will simply go to other sources of information.

Obviously you missed Page 1 and how some of the household-names started to make a statement like that -- According to your statement, Facebook, Amazon and Pinterest didn't grow without showing in Google's results, because people who searched simply went elsewhere to find what they were looking for, which is false.

WebmasterWorld even blocked gBot for months and still continued to grow, which means you're really not telling new users [or anyone else] the truth about things and what can be [has been] done without Google's algo being involved -- I know that may come as a shock to many who are completely drunk from all the Googleade, but it's simply the truth.