Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Pages Dropping Out of Big Daddy Index

Part 3



12:36 am on May 17, 2006 (gmt 0)

5+ Year Member

< continued from [webmasterworld.com...] >

Seems to me that Matt's recent message confirms my theory. We're either all a bunch of moaning idiots with low quality sites with a few innapropriate, spammy links scattered here and there...or...

The more I think about it the more convinced I am that the missing pages problem is being caused by a Backlink/PR issue (see Msg #15).
Tying together all of the evidence from my own experience, and that of others gleaned from the forums, erroneous or out-of-date backlinks would explain all of the missing pages.

The erroneous, or simply out-of-date, backlink information (which we cannot see) leads to insufficient PR (which we cannot see) and hence deep pages are not indexed.

We all know that a "link:www.mysite.com" does not show you the complete picture. But, since Big Daddy, it now shows just a tiny proportion of backlinks. Way less, than it used to show before Big Daddy. Why? Because either the backlink index hasn't been updated (and now dates back to mid 2005), or else because it has been updated, but the update process is buggy. Only a small handful of Google employees know which of these two possibilities is the case.

We know that the missing pages problem cannot be due to any kind of duplicate content filter, as some people are suggesting. If this were the case, then effected sites would see a proportion of their pages disapear. Some would lose 10%, some would lose 40%, and some would lose 95%. But that's not what we see. We see sites losing the vast majority of their pages or else losing no pages at all. The reason effected sites lose such high percentages of their pages is because of the hierarchical nature of a site. The number of pages increases with depth, and the artificially low PRs (based on innacurate and/or out-of-date backlink data) prevents the deeper content from being indexed.

The fact that Big Daddy was kick-started from an index dating back to the middle of last year, not only explains why the backlink data might be stale, but it also explains why ancient pages keep popping up on various data centres.

As further evidence: try a "link:www.mysite.com" and compare it to a search for "www.mysite.com". In my case, the "link:" search shows just 6 results, only one of which is external to my site. The one external backlink probably pre-dates when Big Daddy's index was seeded. The "www.mysite.com" search, on the other hand, finds hundreds of results representing hundreds of internal and external backlinks. Why aren't these showing up in the "link:" search? Is it because "link:" searches are well known for not showing you the complete picture? Or, has that well-known fact simply been obscuring the true cause of all of the problems? Namely, that the backlinks are simply missing from Google's backlink index.

[edited by: tedster at 8:25 pm (utc) on May 17, 2006]


2:44 pm on May 19, 2006 (gmt 0)

10+ Year Member

>> I am generally not a believer in conspiracy theories, but has anyone stopped to consider that this entire BigDaddy fiasco may actually have a motive? It's a very efficient way for Google to test how many webmasters and small businesses will convert into Adwords customers after their sites have been decimated.

It's an interesting theory, and people are searching for explanations about why Google has botched this up so badly.

The problem is, while I'm sure Google wants to get more people into AW, if they make it hard for people to find sites running AS, then they are shooting themselves in the foot.

There's also the anger and trust factor - I have a service that I use AW to promote, and I also have sites with AS on them (the two are seperate I don't actively use AW to promote my AS sites, although sometimes if I'm launching a new site I might run AW for a week or two just to pick up some mindshare and get noticed), and I have two perspectives:

1)As an AW user, I see the problems with my sites, and I have to question whether my AW dollars are being spent wisely - I have the feeling that my Ads will not run on the most relevant sites, and that I am wasting money. If my sites are not being indexed properly, and many others are reporting the same, then I just don't have confidence as an AW user.

2)As an AS publisher, I'm agry that Google has botched this up, and have no desire to keep using AS, and this carries over to AW - why should I give money to a company that screws up like this? Why should I run AS that benefits Google as well, when there are a lot of other advertising programs out there?

It makes no sense why Google has botched this up so badly - it benefits them to index as much as possible, relevant sites that are going to help with AW/AS.

It pisses me off that the spam sites in my niches are doing just fine in the indexes while I've been hammered.

Heck, maybe because the spam sites are spam sites and geared towards getting people to click an ad and leave, rather than keeping people around and informing them like many of us do, that the spam sites are more valuable to Google (or at least to Google's indexes).

I'm not saying that Google (the company and people that make up Google) wants spam sites, I'm saying that Google's software is keeping the spam sites around because they are better for AS.


2:45 pm on May 19, 2006 (gmt 0)

5+ Year Member

Frankly, I hesitate to post this type of thing because, well, Google may look at it as being a post that trys to influence their thinking. God knows they have staff that keep an eye on these posts.

But, that said, I'll flap my gums anyway because I've got something to say.

I have never before heard the amount of user complaints about Google as I have heard over the past couple months from general search users.

I've heard dozens of people complain that Google search results have been for crap with Google lately.

Somehow, I think Google is relying on feedback to gauge their success. But, the people I know are not going to take the time and send Google am e-mails complaining about the fact they can not find what they are looking for. Rather, they just move on to "brand X".

Hmmm, perhaps Google will be smart and conduct a thorough survey of users. If they do that, I have no doubt that they will find thier percentage of satisfied users is declining faster than George Bush's ratings.

For me, I no longer consider the words of MC and GG to be credible. They have lost their credibility with me simply because ... well, most of their words have been reactionary instead of ahead of the curve. So, this tells me that they have no real clue what is going on.

And, I suspect, the major majority of Google engineers are operating on a wing and a prayer right now from the looks of it.

In the end ... we need to watch the money. If webmasters do not do the AdWords thing, the reduction of income for Google will force them to change ... er, rather, stockholders will demand they change.

And then, up in the crystal boardroom at the Plex, the men in the suits and ties will respond and quietly demand that things be tweeked once again.

Indeed, follow the money.


3:05 pm on May 19, 2006 (gmt 0)

I've heard dozens of people complain that Google search results have been for crap with Google lately.

LOL, send them to MC's blog....perhaps that's what he is really after. The entire web's attention to his blog and wired arrogant sense of humor. Contrary to you, I have a strange feeling he has a lot to do with the recent sensless flop.


3:26 pm on May 19, 2006 (gmt 0)

I've just checked out MC's blog and I can't believe what I see. It would appear that he has gone 'Tony Blair' on us and is telling us exactly how to conduct our lives ...

e.g. "- If you’re sending out an email with links in it, check your links by clicking on them before you send that email."

Thanks Matt, when you can get a relavent website to the top of your listings, maybe then I'll start taking advice on how to live my life from you.

God, I'm feeling SICK now!

King of all Sales

3:45 pm on May 19, 2006 (gmt 0)

5+ Year Member

We run a fairly extensive Adwords thing with 9 campaigns and bucketloads of keywords and we spend quite a bit of money. Yet we are unable to have any meaningful relationship with Google as we throw cash at them.

As much as Google thinks they can change the way business is done - completely automate it - it will always be about humans interacting with other humans. Some things never change.

Google almost seems to have a palpable disdain for any sort of direct contact with the people who pay their bills and that translates to arrogance.

With the latest round of Big Daddy which hurt a little and the news that Cutts is taking a 6 week vacation (practically unheard of here in America), we have analyzed our adwords and decided we can trim the fat, reduce our spend at between 10 & 20% and the only effect will be that we increase our ROI.

Now that Y and MSN are on the move and have very viable ad programs, we will give the money to them. We are not trying to teach Google a lesson other than that you shouldn't announce to your customers that you are taking an extended vacation on their dime.

We just think that since they have done absolutely nothing to secure our loyalty, we don't need to put so many eggs in their basket.


4:20 pm on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member play_bach is a WebmasterWorld Top Contributor of All Time 5+ Year Member

After reading here about other sites losing pages daily, I decided to start counting one of my sites on Saturday using site:example.com (this site should have 550+ pages indexed!).

Saturday = 113
Sunday = 103
Monday = 96
Tuesday = 88
Wednesday = 87
Thursday = 82
Friday = 75

At this rate, I figure my site will be all but gone by June. :-(


4:31 pm on May 19, 2006 (gmt 0)

10+ Year Member

A fundamental element in Google traffic doesn't seem to have been raised here. I checked back over years of Adsense data and one fact is very clear. Clicks from Google paid ads are currently converting at an all time low, down to 15-20% of their level 2 years ago. Yahoo are holding up much better at about 90% of their previous level.

If anyone thinks that driving more companies to have to pay for clicks will be good they are dreaming.

Whatever Google is up to we have passed the point where sending any money to Google is profitable. We have also seen many other companies drop out of our niche. Now supply and demand theory says that will reduce the cost and get us back in but no amount of discounting will compensate for near zero conversion.

Either there is massive click fraud or surfers are getting jaded at being shoved to page after page of dubious content with yet more Google ads to click.

Something in the heart of the Google beast is very very sick right now


4:28 pm on May 19, 2006 (gmt 0)

5+ Year Member


my site have a good PR. and get in 53 pages in indexing. now my site have only 10 pages in google cache.

I saw a PR 4 of one page. but in no cache in google of that page.

please clarify of it.



4:56 pm on May 19, 2006 (gmt 0)

10+ Year Member

King of all Sales is right, it's not about punishing Google, it's about evening the playing field. Without Google my business would not be in the position it's in in the first place, but Yahoo and MSN are catching up. I look forward to the day when all three SEs are neck and neck, as then we will have true competition in the search market. Right now it's a Google monopoly, and that's never good.

natural number

4:56 pm on May 19, 2006 (gmt 0)

10+ Year Member

Well, I didn't really follow this thread for a while. Then I took the time to do the old "site:" command.... and great balls of fire, google only shows two pages from my actual site! The rest of the links are just links that have picked up from around here and there.


5:04 pm on May 19, 2006 (gmt 0)

5+ Year Member

I saw a PR 4 of one page. but in no cache in google of that page.

any body answer of it

natural number

5:09 pm on May 19, 2006 (gmt 0)

10+ Year Member

Earlier I said I only had two sits appear with the "site:" command, I must have did it wrong. I'm fine as far as that's concerned. Sorry for the hubbub I may have caused.


5:11 pm on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member

70 pages indexed on a small site of mine yesterday, now there is only 20. Why index and cache a page if you are going to drop it. It has been that way the past three weeks with this site. It is spread across all data centers. One day 70, the next day is 20.


5:15 pm on May 19, 2006 (gmt 0)

10+ Year Member

Why consume our bandwidth just to do that?


5:16 pm on May 19, 2006 (gmt 0)

10+ Year Member

[I am generally not a believer in conspiracy theories, but has anyone stopped to consider that this entire BigDaddy fiasco may actually have a motive? It's a very efficient way for Google to test how many webmasters and small businesses will convert into Adwords customers after their sites have been decimated.]

I turned off all our clients Adwords and switched over to Yahoo PPC and our own sites, this Google SERP changing is BS. Nothing at all makes sense anymore with Google.

Trust me Yahoo PPC conversions are always better and more serious that Googles! (MUCH BETTER conversions)


5:19 pm on May 19, 2006 (gmt 0)

10+ Year Member

I personally don't believe in PPC advertising. I tried it for my site when it was new and it didn't bring in any business. People will always prefer natural search results over paid ads given the choice. All the problems with click fraud made me drop my adwords campaign completely. Of course, if your customers have money to burn that's another story.


5:56 pm on May 19, 2006 (gmt 0)

10+ Year Member

>>this entire BigDaddy fiasco may actually have a motive?<<

Is it a fiasco? Or is it the best attempt at "cleansing" Google has ever made?

You cant ignore MC's comments in his "timeline" entry, you dont like what he saying but you defintely cant ignore it either. The big players on these boards havent said too much about the issues yet because thats usually the way when something big stirs - sensible, knowledgeable people take time to absorb the info and analyse before committing to an opinion.

Already there are threads about doing away with links pages and purging crappy recips from sites. This isnt going to save the more prolific recip based / link monkey sites from a slow indexing death.

Tell you what tho - Ive thought more this week about link quality than ever before. Content is king they say - if only.

Great job MC


6:06 pm on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

im sorry but this link quality tangent is really a red herring....


7:02 pm on May 19, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Personally I’m skeptical of anything Cutts and GG say now. I mean really when you start quoting people who are talking about giving quality time to “kitty kats” you need to look in the mirror and reshape your priorities in life.

Plus where’s that GG with his “content is king” crapola. Not pushing Adsense anymore?

I love those words “cleansing”, "purging", and “quality” also. If history has taught us one thing whenever “censorship” is involved those words never fail to appear. The problem is there’s always an ulterior motive behind those words.


11:09 pm on May 19, 2006 (gmt 0)

5+ Year Member

link more!


1:16 am on May 20, 2006 (gmt 0)

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

It now looks like I'm completely on the downslide. I'm losing roughly a page an hour. At this rate I'll be gone in a month.


1:41 am on May 20, 2006 (gmt 0)

5+ Year Member

Hi new to this forum...but have been on others for years. I have a travel site that has around 120,000 pages before all this started. I have seen no drop in my pages but 30% of them don't show in the G index.

If I enter any of the missing URLs in G they show up and are being updated but missing from the index.

Some of the missing pages are starting reappear but its very slow coming. All of my missing pages are at the bottom of my site 4 steps down from the homepage and most are PR 2 pages. All of my pages 1, 2 or 3 steps from the homepage are there!

When I check other travel sites some are having the same problem and others are not!

I really think that this a just a G screw up and nothing more and they are trying to fix it.



2:31 am on May 20, 2006 (gmt 0)

10+ Year Member

Recently lost about 60 pages from 130.

All they are far from the root. Most of the pages are new.

So the main reason may be indeed that Google has not enough computational power to index all pages and has to select.

Possible reasons that triggered the drop.

1.Recently I used AdWords without real company, just to research keywords. Might be Google tries to force me to pay for adwords (I observed similar effect two years ago with the same site).

2.My software was recently cracked and I got the links from the cracker sites that suggested downloading my soft. Might be Google do not like to waste the precious index resources for the site that has suspicious inbound links.

Preliminary signs that something is wrong,

1.In sitemap statistics Google began to report that it has problem to index a part of my site.

2.In sitemap statistics Google began to show significantly smaller number of the pages with high PR.

Do you also observe similar pattern?



4:29 am on May 20, 2006 (gmt 0)

5+ Year Member

Lets hope it is a Google cock-up as more and more people will be turning to the dark side I would have thought.


5:51 am on May 20, 2006 (gmt 0)

10+ Year Member

By the time they fix it their user base is going to be jumping ship. If this new infrastructure is supposedly so great why are there so many problems?


5:51 am on May 20, 2006 (gmt 0)

10+ Year Member

Did anyone notice link to the cache is missing on the SERP's? Is it just me?


6:00 am on May 20, 2006 (gmt 0)

10+ Year Member

Cache is working for me...


8:45 am on May 20, 2006 (gmt 0)

5+ Year Member

My numbers going up again for older sites.

But for new sites, there are only pages in indexed that are 1 click from homepage, all other i have lost from index, thats interesting.


9:45 am on May 20, 2006 (gmt 0)

10+ Year Member

Before this started I could not stop Google from indexing anything, be it with metatag or robots.txt. And there were only a couple of supplementals.
No my site went from 500 indexed pages to 12 plus about 50 supplementals with a totally irrelevant cache from early to mid 2005.
Surely this can't be the new improved Google?


11:12 am on May 20, 2006 (gmt 0)

5+ Year Member

Does MCutts takes a 6 weeks holiday just for fun?,or he needs some energy of some updates that will come after 6 weeks?
This 213 message thread spans 8 pages: 213

Featured Threads

Hot Threads This Week

Hot Threads This Month