Forum Moderators: Robert Charlton & goodroi
Seems to me that Matt's recent message confirms my theory. We're either all a bunch of moaning idiots with low quality sites with a few innapropriate, spammy links scattered here and there...or...
The more I think about it the more convinced I am that the missing pages problem is being caused by a Backlink/PR issue (see Msg #15).
Tying together all of the evidence from my own experience, and that of others gleaned from the forums, erroneous or out-of-date backlinks would explain all of the missing pages.The erroneous, or simply out-of-date, backlink information (which we cannot see) leads to insufficient PR (which we cannot see) and hence deep pages are not indexed.
We all know that a "link:www.mysite.com" does not show you the complete picture. But, since Big Daddy, it now shows just a tiny proportion of backlinks. Way less, than it used to show before Big Daddy. Why? Because either the backlink index hasn't been updated (and now dates back to mid 2005), or else because it has been updated, but the update process is buggy. Only a small handful of Google employees know which of these two possibilities is the case.
We know that the missing pages problem cannot be due to any kind of duplicate content filter, as some people are suggesting. If this were the case, then effected sites would see a proportion of their pages disapear. Some would lose 10%, some would lose 40%, and some would lose 95%. But that's not what we see. We see sites losing the vast majority of their pages or else losing no pages at all. The reason effected sites lose such high percentages of their pages is because of the hierarchical nature of a site. The number of pages increases with depth, and the artificially low PRs (based on innacurate and/or out-of-date backlink data) prevents the deeper content from being indexed.
The fact that Big Daddy was kick-started from an index dating back to the middle of last year, not only explains why the backlink data might be stale, but it also explains why ancient pages keep popping up on various data centres.
As further evidence: try a "link:www.mysite.com" and compare it to a search for "www.mysite.com". In my case, the "link:" search shows just 6 results, only one of which is external to my site. The one external backlink probably pre-dates when Big Daddy's index was seeded. The "www.mysite.com" search, on the other hand, finds hundreds of results representing hundreds of internal and external backlinks. Why aren't these showing up in the "link:" search? Is it because "link:" searches are well known for not showing you the complete picture? Or, has that well-known fact simply been obscuring the true cause of all of the problems? Namely, that the backlinks are simply missing from Google's backlink index.
[edited by: tedster at 8:25 pm (utc) on May 17, 2006]
It's an interesting theory, and people are searching for explanations about why Google has botched this up so badly.
The problem is, while I'm sure Google wants to get more people into AW, if they make it hard for people to find sites running AS, then they are shooting themselves in the foot.
There's also the anger and trust factor - I have a service that I use AW to promote, and I also have sites with AS on them (the two are seperate I don't actively use AW to promote my AS sites, although sometimes if I'm launching a new site I might run AW for a week or two just to pick up some mindshare and get noticed), and I have two perspectives:
1)As an AW user, I see the problems with my sites, and I have to question whether my AW dollars are being spent wisely - I have the feeling that my Ads will not run on the most relevant sites, and that I am wasting money. If my sites are not being indexed properly, and many others are reporting the same, then I just don't have confidence as an AW user.
2)As an AS publisher, I'm agry that Google has botched this up, and have no desire to keep using AS, and this carries over to AW - why should I give money to a company that screws up like this? Why should I run AS that benefits Google as well, when there are a lot of other advertising programs out there?
It makes no sense why Google has botched this up so badly - it benefits them to index as much as possible, relevant sites that are going to help with AW/AS.
It pisses me off that the spam sites in my niches are doing just fine in the indexes while I've been hammered.
Heck, maybe because the spam sites are spam sites and geared towards getting people to click an ad and leave, rather than keeping people around and informing them like many of us do, that the spam sites are more valuable to Google (or at least to Google's indexes).
I'm not saying that Google (the company and people that make up Google) wants spam sites, I'm saying that Google's software is keeping the spam sites around because they are better for AS.
But, that said, I'll flap my gums anyway because I've got something to say.
I have never before heard the amount of user complaints about Google as I have heard over the past couple months from general search users.
I've heard dozens of people complain that Google search results have been for crap with Google lately.
Somehow, I think Google is relying on feedback to gauge their success. But, the people I know are not going to take the time and send Google am e-mails complaining about the fact they can not find what they are looking for. Rather, they just move on to "brand X".
Hmmm, perhaps Google will be smart and conduct a thorough survey of users. If they do that, I have no doubt that they will find thier percentage of satisfied users is declining faster than George Bush's ratings.
For me, I no longer consider the words of MC and GG to be credible. They have lost their credibility with me simply because ... well, most of their words have been reactionary instead of ahead of the curve. So, this tells me that they have no real clue what is going on.
And, I suspect, the major majority of Google engineers are operating on a wing and a prayer right now from the looks of it.
In the end ... we need to watch the money. If webmasters do not do the AdWords thing, the reduction of income for Google will force them to change ... er, rather, stockholders will demand they change.
And then, up in the crystal boardroom at the Plex, the men in the suits and ties will respond and quietly demand that things be tweeked once again.
Indeed, follow the money.
I've heard dozens of people complain that Google search results have been for crap with Google lately.
LOL, send them to MC's blog....perhaps that's what he is really after. The entire web's attention to his blog and wired arrogant sense of humor. Contrary to you, I have a strange feeling he has a lot to do with the recent sensless flop.
e.g. "- If you’re sending out an email with links in it, check your links by clicking on them before you send that email."
Thanks Matt, when you can get a relavent website to the top of your listings, maybe then I'll start taking advice on how to live my life from you.
God, I'm feeling SICK now!
As much as Google thinks they can change the way business is done - completely automate it - it will always be about humans interacting with other humans. Some things never change.
Google almost seems to have a palpable disdain for any sort of direct contact with the people who pay their bills and that translates to arrogance.
With the latest round of Big Daddy which hurt a little and the news that Cutts is taking a 6 week vacation (practically unheard of here in America), we have analyzed our adwords and decided we can trim the fat, reduce our spend at between 10 & 20% and the only effect will be that we increase our ROI.
Now that Y and MSN are on the move and have very viable ad programs, we will give the money to them. We are not trying to teach Google a lesson other than that you shouldn't announce to your customers that you are taking an extended vacation on their dime.
We just think that since they have done absolutely nothing to secure our loyalty, we don't need to put so many eggs in their basket.
Saturday = 113
Sunday = 103
Monday = 96
Tuesday = 88
Wednesday = 87
Thursday = 82
Friday = 75
At this rate, I figure my site will be all but gone by June. :-(
If anyone thinks that driving more companies to have to pay for clicks will be good they are dreaming.
Whatever Google is up to we have passed the point where sending any money to Google is profitable. We have also seen many other companies drop out of our niche. Now supply and demand theory says that will reduce the cost and get us back in but no amount of discounting will compensate for near zero conversion.
Either there is massive click fraud or surfers are getting jaded at being shoved to page after page of dubious content with yet more Google ads to click.
Something in the heart of the Google beast is very very sick right now
my site have a good PR. and get in 53 pages in indexing. now my site have only 10 pages in google cache.
I saw a PR 4 of one page. but in no cache in google of that page.
please clarify of it.
Ravi
any body answer of it
I turned off all our clients Adwords and switched over to Yahoo PPC and our own sites, this Google SERP changing is BS. Nothing at all makes sense anymore with Google.
Trust me Yahoo PPC conversions are always better and more serious that Googles! (MUCH BETTER conversions)
Is it a fiasco? Or is it the best attempt at "cleansing" Google has ever made?
You cant ignore MC's comments in his "timeline" entry, you dont like what he saying but you defintely cant ignore it either. The big players on these boards havent said too much about the issues yet because thats usually the way when something big stirs - sensible, knowledgeable people take time to absorb the info and analyse before committing to an opinion.
Already there are threads about doing away with links pages and purging crappy recips from sites. This isnt going to save the more prolific recip based / link monkey sites from a slow indexing death.
Tell you what tho - Ive thought more this week about link quality than ever before. Content is king they say - if only.
Great job MC
Plus where’s that GG with his “content is king” crapola. Not pushing Adsense anymore?
I love those words “cleansing”, "purging", and “quality” also. If history has taught us one thing whenever “censorship” is involved those words never fail to appear. The problem is there’s always an ulterior motive behind those words.
If I enter any of the missing URLs in G they show up and are being updated but missing from the index.
Some of the missing pages are starting reappear but its very slow coming. All of my missing pages are at the bottom of my site 4 steps down from the homepage and most are PR 2 pages. All of my pages 1, 2 or 3 steps from the homepage are there!
When I check other travel sites some are having the same problem and others are not!
I really think that this a just a G screw up and nothing more and they are trying to fix it.
Richard
All they are far from the root. Most of the pages are new.
So the main reason may be indeed that Google has not enough computational power to index all pages and has to select.
Possible reasons that triggered the drop.
1.Recently I used AdWords without real company, just to research keywords. Might be Google tries to force me to pay for adwords (I observed similar effect two years ago with the same site).
2.My software was recently cracked and I got the links from the cracker sites that suggested downloading my soft. Might be Google do not like to waste the precious index resources for the site that has suspicious inbound links.
Preliminary signs that something is wrong,
1.In sitemap statistics Google began to report that it has problem to index a part of my site.
2.In sitemap statistics Google began to show significantly smaller number of the pages with high PR.
Do you also observe similar pattern?
Vadim
But for new sites, there are only pages in indexed that are 1 click from homepage, all other i have lost from index, thats interesting.
