Google banned my sites. Should I ban Googlebot?

[1script]

I know, I know, this comes across as an "eye for an eye" type of deal but what I'm actually looking for is more like:"what would Je..., I mean, tedster do ?" and has a purely practical application.

Having managed to have most of my money earning sites banned in one day (long story [webmasterworld.com]), I am looking at some austerity measures the likes of which I never had to implement before. It includes elimination of hosting accounts and aggregation of sites on a small number of VPSes which is where the question of the tremendous burden that Googlebot places on any site comes in.

Sure, there are other bots that sap bandwidth but Gbot deserves an honorary mention. It has always been the largest consumer of bandwidth and CPU but now, with real people from Google gone, the amount of resources it consumes compared to the benefit to me (ZERO) is simply ridiculous.

Despite the ban, Googlebot continues to pummel all my banned sites as if nothing happened. In fact, the amount of URLs in web index in WMT continues to grow even after the ban.

So, what would you guys do: let Googlebot crawl the sites unchecked despite them bringing no traffic or ban Googlebot (either through robots.txt or possibly even firewall) which will then enable you to save money on smaller server/bandwidth.

There is another facet to this: I am using AdSense on most of my newly banned sites. Since Mediapartners-Google bot shares the data, Google will still have a pretty good idea about new URLs on my sites even though Googlebot itself is banned. If and when any of my sites get un-banned (does that even happen?) I can re-enable Googlebot access.

So, does anyone think it may be a good idea to mess with Googlebot access?

Cheers!

[dstiles]

I wish I could kill googlebot but my clients won't let me. :)

I will say, though, that googlebot isn't the top bot on my server. That slot belongs to yahoo, and always has done. Since yahoo now gets its SERPS from bing it could be advantageous IN YOUR CASE to block yahoo slurp. Whether that is a good idea on other sites I'm not sure.

[wheel]

Congrats on having the guts to get banned, condolences on being banned.

Personal opinion, I'd ban Googlebot in a heartbeat. It's as simple as they can crawl and cost if they send me traffic. No traffic, no crawl. And I'd IP ban them, not just robots.txt.

[onepointone]

On my banned sites that I've left up, googlebot continued to hammer them for a couple of months, if I recall...

But after that, they kind of banned themselves. Now i get 0 hits/month. Sometimes 1 hit.

[MrFewkes]

Ban the gbots - and start up a new site which actively promotes that other people do the same.

[ascensions]

Ban that loser! He sucks! Bots are the beast!

[Sgt_Kickaxe]

Infact yes, you should absolutely ban googlebot if your site is not indexed. There is absolutely no reason left to allow googlebot to load your every page after a ban. You wouldn't allow it from any other site that sends no traffic, Google is no different in your example.

I think the days of rolling the red carpet out for Google may be nearing an end as they shift away from being a pure search engine.

If you're inclined to do so a small "this site is Google free" button may inspire other webmasters to see that there is life outside of Google.

[1script]

Wow, guys, I had no idea banning Google is so popular in this establishment!

"this site is Google free" sounds like a great fun but I'm not sure I'm quite there yet. In fact, I would love to see the sites reinstated at some point. So, if I knew for a fact that they banned me for good, I would ban them back in a heartbeat. But I want to give my sites a fighting chance, small as it may be. Some of them represent 5+ years of my time and loads on money invested, well, you get the idea.

So, I'll have to rephrase the question: if you didn't want to burn bridges for Google, would you still ban Googlebot for the duration of Google ban, assuming the ban is temporary (I know, it's one heck of an assumption!). Will Google's inability to crawl the site while the reconsideration requests are making their way through the system hinder the site's chances to recover? Has anyone been through this before and got the site reinstated?

[onepointone]

I'd wager that the chances of getting reinstated would go from slim to none by banning googlebot.

And if your 'SiteA' was banned by being associated with your 'SiteB' (with something nasty on it), then both sites would need to be open for g to crawl.

JMO

[MrSavage]

Think of Google as a person. For them to like you, they must trust you. Right now they penalized you. Hiding behind a bot block sends the wrong message to them. If you care to bring them back from the dead you won't be making it easier or quicker. Let the frustration pass for a week or two then reconsider the plan.

[FranticFish]

I think you need to ask yourself "Are Google and I friends who have had a misunderstanding, or are we no longer friends?"

Bans can last for months or years. How long can you afford to try to salvage the relationship?

Maybe PM MadScientist for some advice too. Earlier this year I think he launched his first 'no Google' site i.e. a site deliberately built with absolutely no Google SEO considerations, doing exactly what he wanted to for users and marketing it as if search engines didn't exist. It was an experiment rather than a necessity, but still...

[MrFewkes]

I think that a "This site is google free" button would be a very truthful act in terms of a reflection of how millions of site owners feel.

I admit - I tend to be a bit more aggressive than that - but I like the idea as its aggressive - yet subtle if that makes sense.

I think I would put something like "For Incomplete Results - Use Google" - particularily as the button spread it would be effective.

FranticFish - I think that for millions of site owners - the answer to your question would be "We are no longer friends" - for various reasons. Sadly - for millions of other site owners - the answer is still I love google and google loves me.

I may be an example of a site owner with both answers - as I have many sites. Some google clearly hates, some it loves (mainly the ones which I dont make any money on!)

Its tricky and I dont think theres an answer - until they push it too far by shunting natural serps down so much that users will see they are nothing but a spam advert server these days.
Once that happens - I am hoping to see another altavista scenario.

Goto.com/overture never really took off like google - but now that google is overture with a few natural serps at the bottom - hopefully people will start moving to other search engines.

I hope this post doesnt sound aggressive or anything - im just observing what happens when the engine upsets webmasters - nothing - but when it upsets the users - well thats another story history tells us.

[deadsea]

I'd think about letting googlebot crawl only a small portion of the site. That way your hosting expenses due to googlebot would be minimal and you would notice if you ever started getting traffic on the pages that are still crawled.

[netmeg]

#1 I wouldn't let anyone else tell me what I should or shouldn't do with my site (re "Google free" buttons and whatnot)

#2 I would NOT ban Googlebot unless and until I was dead certain I had exhausted possibility of getting the site back in the index.

#3 I'd beg for or hire other pairs of eyes to look over the site for things I may have missed - we always miss something, specially if it's our own site

#4 I'd start screaming bloody murder in places where there are actual Google reps who might hear me, like the Google help forum, twitter, Google+, friends with inside connections, etc.

Then and only then I might consider banning the bot.

But that's just me.

[mrguy]

As others have said, if you're sure the domain is toast with no hope of recovery, then why would you not ban them?

At that point all Google is doing is sucking up your bandwidth and not giving you anything in return.

[pageoneresults]

There is another facet to this: I am using AdSense on most of my newly banned sites.

Remove the AdSense or trim down to one or two units and see what happens. You say Googlebot is crawling, which I'd say is a good sign. Anytime I see AdSense injected into something of this nature, that's the first thing to address. Did you possibly overdo it with the number of and/or placement of AdSense units?

I don't think I'd ever find a reason to ban Googlebot. You definitely need to do some hand holding once Googlebot visits but banning the bot would not be the right thing to do. You could recover at anytime and if you banned the bot, you wouldn't know.

If the site in your profile is any indication of what the other sites look like in terms of AdSense placement, there's your reason why they've been nuked. :(

[1script]

Anytime I see AdSense injected into something of this nature, that's the first thing to address. Did you possibly overdo it with the number of and/or placement of AdSense units?

I might have of course. I run forums and forums don't convert. In fact, the more people use then the less they convert because of the banned/ad blindness. If you don't have ads above the fold, forget it - you might as well go get a (different) job.

So, what is this anti-ad champaign (not referring to your post, I've seen it in enough places elsewhere)? A push to have more sites run as a hobby?

How many AdSense publishers here can say they haven't received a message from Google just recently titled something like:"You’re missing 1092 opportunities to earn more!". You open it and see something like:

At least 546 of the pages on example.com are running less than three ad units. Adding any of our top performing units (336x280, 300x250, 728x90 or 160x600) to prominent sections of your pages will substantially increase your ad impressions and overall revenue. (emphasis mine)

So, which is it? More ads or less ads? I do experiment with ad placements but I always end up with placements straight from Google's own heat map.

I know, Matt Cutts does not report to or may not even ever cross paths with the AdSense team. But one company-wide policy would have been nice. For example, a note from the AdSense team saying something like: "forget about that message pushing for more ads we sent earlier. We've had an internal discussion and decided we are for better looking pages now. Please remove one of two units from the prominent sections of your pages"

So, yeah, all of this sucks but I think I came up with a solution for dealing with Gbot after all: I'm going to use the Crawl Delay directive and set it at a very high level, something like:

User-agent: Googlebot
Crawl-delay: 30

They're getting 2 pages in 3 seconds right now. If I slow them down 20 times, that would do it for me. And they won't feel bad about being banned for banning me :)

[MrSavage]

The unfortunate aspect is whether or not you as a person (anything associated with your Google accounts) is now tarnished. In other words, if this was a violation against multiple sites, then it's comes back to the trust factor. If the penalty seems more personal, as in no real obvious penalty, then it's more serious. Depending on what is really the issue, I would be considering ditching accounts. Again, that's just me. We all want to think that penalties are not something that can tarnish you as a person, but there is no proof either way. If you are a troublemaker, why wouldn't Google put you on some sort of list and keep track of you? It makes sense. Not saying you're one of those "bad guys" as Cutts likes to say.

[pageoneresults]

So, which is it? More ads or less ads? I do experiment with ad placements but I always end up with placements straight from Google's own heat map.

And that is exactly what has gotten most folks into trouble these days. Too many ads above the fold, placed in areas where navigation is normally expected, yada, yada, yada. The Google Search Team and the Google AdSense Team operate independently. One is concerned about more revenue, the other about improving search. Unfortunately the two don't work well together.

Those notifications from AdSense come with a long rope attached to them. Look at many of the sites that have been part of the ongoing Pandaemic. They too followed the AdSense advice and look where it got them.

I've never been fond of those recommendations from AdSense, they just go against usability guidelines and reek of greed. I never understood why someone would put an AdSense unit in the left hand nav above the fold right above their on site nav. "The heat map says so." No, the heat map indicates where a person's eyes are likely to travel when first landing on your page. Do you really think the first thing they should see are ads?

Just how many ads have been clicked by mistake due to the placement of ad units? Google knows that stuff and if your site crosses that threshold, Pandamonium ensues.

[MrSavage]

If ads were reason for a penalty type situation, 50% (slight exaggeration) of sites would be off Google right now. I'm not seeing that personally.

On a side note, ever notice search pages? Where are those ads? Yeah. Where is the content? Yeah, way below the ads. We could argue relevance but...

What about Google Adsense for Search. How about that experience for the end user? Not exactly pleasant user experience imo. But that's ok.

I'm just saying, ads are a tipping point for penalties? We could argue about Panda but I can find so many counter examples that argument would go nowhere.

[netmeg]

If he's completely out of Google, it ain't ads.