I've banned Googlebot

What Will Happen Now?

[Iguana]

Last Saturday 8th July I disallowed Googlebot in my robots.txt for one of my sites. The only effect so far seems to have been that my description in the snippet has reverted to my DMOZ entry and my page count in Google has gone down from 472 to 469.

I expected to see something a bit more radical happen. Can anyone predict what will happen? Will my pages go Supplemental or even disappear from the index?

[doc_z]

Your page will be completely removed from the index. A detailed description is given here. [webmasterworld.com]

[ncgimaker]

Iguana, wait a little longer, we kicked G off one of ours and it took a week for the site to disappear but then that did conincide with an update.

[edited to correct explanation]

[Iguana]

Doc_z

I havent used the Google removal tool and don't intend to - purely robots.txt. Can I still expect to me my complete removal?

[ncgimaker]

Iguana,

We didn't use the removal tool like doc_z, just robots.txt, all our pages disappeared except for the link url into the site from external pages (without text or snippet). So I would expect the same to work for you.

[Aaz7]

hehe... It's time to sell Google's shares... :)

[suidas]

I've stopped calling my friends and when they call I swear at them and hang up. What will happen now?

[Iguana]

Thanks ncgimaker, useful to know I can use this rather than a removal

As you say, I may have to wait till an update (another 3 pages less in Google today) for everything to disappear properly.

[doc_z]

I would expect that all information are removed from the index (even URL only entries). The time until all pages are removed might depend on the crawling frequency.

[andrea99]

I'm curious to hear the reasoning behind the ban. Too much traffic? A private site with personal secrets you only want to reveal to a select few?

[mack]

I realize there has to be a good explination, but I have to admit I am curious. What made you decide to ban Googlebot?

Was Google simply not delivering traffic, was the bot using up to much bandwidth or another reason?

Mack.

[Iguana]

The reason for banning Googlebot is pretty simple - it can't hurt. For this particular site I rank between #13 and #70 for my own name(!). Therefore I get about 1-3 visitors per day from Google. I've done all the getting rid of 302's, sorting out non-www issues (not that there was more than 1 link to non-www), emailing GG on the update group.

There's a possibility that Google is looking at related sites whether through history of links or even whois. Maybe by taking this site out of the index the other site that makes a bit of money might return to it's former position. Also maybe my blog on blogspot (named after the removed site) may rank on the first page so that bands looking for someone to review their CDs will find a link to my site. As I say, it can't hurt to try. I don't want to use the remove URL because I may want to remove the ban in a month or so.

The last time I took this action was with with Yahoo about 18 months ago. Within 10 days I was ranking really well (as compared to last on exact string searches). Okay, I know it was just coincidence then, but when logic fails, choose the irrational.

[Kelowna]

I blocked one of my sites using robots.txt thinking that once it was removed I would then resubmit. Kinda like trying to start with a fresh slate.

The problem is that the site will be removed for 6 months, which I never new at the time, now I have another 3 months before the site is indexed again(hopefully).

The only reason I did this was there was another domain that hijacked the site with the old redirect trick so I thought this would put an end to that as google was taking forever to fix the hijacking problems.

I found out at New Orleans that using robots text to get a site removed does so for 6 month's so it should be used with caution.

[Iguana]

Kelowna

I assume you changed robots.txt and then used the Google remove URL. I'm not going to remove the URL through that service, so I assume I'll be able to unblock myself at a later date

[Kelowna]

yes you are right, I did use the removal tool. I just thought people should know that the sites get removed for a 6 month period.

[Iguana]

I've just checked my stats for yesterday and today and there's been a huge leap in traffic to my main site (not the one I've banned Googlebot from) - as in 400% increase in page impressions and nearly all originationg from a Google search referral. Yahoo was giving me nearly twice as much traffic as Google but now Google is giving me 4 times the traffic that Yahoo is.

Has there been a major Google update? Not judging by the threads here on WW.

I've just checked a number of searches I always use to judge my sites progress - and I am back in top position for each "widget words" and on the front page for "widget" - a position I last held in August last year. Earlier this week I didn't even appear for "widget words".

What conclusions can I reach at this early stage? Well, the experiment has been successful - something about one of my sites was dragging another down. I did have a time when I linked to my main site on every page (actually liked to the 4 main index pages on my main site so it was a bit excessive) There are pages on both sites that present the same information in different ways so there may be a duplicate content penalty of some sort. It certainly seems to me that this last 10 months of Google 'filtering' has been based on links (and Google IS a search engine ranking system based on links)

Loads of questions still, the site I banned Googlebot from is now up to #4 for its own name. Another site i have hasn't returned to its former position.

Personally I'm having a drink tonight - Czech lager.

[CathyM]

Iguana, That's good news!
I had a similar situation last year. I bought a new URL to replace an old one. I temporarily had similar content on both sites and inter-linking. They both started to rank really well and then the new one slipped away to nothing. I banned Googlebot on the old one with a noindex, nofollow and the new one came back in ther SERPs within a few days. I experimented doing this a few times to make sure it wasn't an anomaly and always got the same results. I eventually left the no index, no follow on the old one for a few months and re-deployed it with completely different content (realizing the value of an old site because of sandboxing of new sites). I later took off the ban and they have both continued to rank very well for their respective content.

[Iguana]

Thanks CathyM - you've also given me hope that I can allow Googlebot back to the site I banned it from in a month or two and maybe it will get decent ranking as well