Forum Moderators: Robert Charlton & goodroi
On one of my searches this morning, I found a site with an expired domain above my site in the results. That just ain't right!
[edited by: tedster at 5:58 am (utc) on Nov. 8, 2006]
Meanwhile whats the bl@@dy point? - have the serps improved - NO, junk is rising to the top and having a field day as a result
Are the serps more relevent - NO junk ranking well as a result of this major blunder
Did it need fixing? - NO - should have left it well alone in the first dam place!
Whats this achieved - Nothing but worse serps and the stuffing of a number of quality authority sites that have supported google pooring millions into adwords!
Do we deserve better than this? - YES
The problem I am hearing for affected domains is different: urls that are still actively being served are being "eaten away" from the main indexes and into the supplemental index -- until there's no url left on a site: search that ISN'T supplemental but the Home Page. As supplemental, these pages will rarely rank for anything except oddball searches -- even though they were ranking beautifully in the very recent past.
I dont think this is exactly what is happening. It is not that your actively served pages are being "eaten away" from the main indexes and into the supplemental index, but rather that these pages are being dropped from the index altogether. Eventually you are left with only your home page as a legitimate entry and then the rest being supp pages. At this point I can not find a single active url from my older pages (that are still visible here: 66.102.9.104 in the new index. They simply have been dropped and all I am left with is homepage and supps. From what I understand..that is the bug
All these pages though, to this day, are still being actively crawled by google.
I have four site first one to suppleMENTAL hell had no 301 then today site with 301 did the same so im not sure it makes any difference.
For each page in the index, there should only be one entry or else Google wouldn't know which page to serve. If you're seeing old cache dates, that because that is the only page in that particular index. I have some theories why Google holds onto pages forever, but that is just my guess.
My suggestions are these (this is what I did):
Look at the supplementals and figure out how they got there in the first place. The examples from my website and the examples folks sent me are all spawned from "sloppy" webmastering. I didn't 410 a page when I should have or I switched naming conventions and didn't redirect right away to the new page. I didn't block new pages from spawning using robots.txt. I didn't return a 404 when a page didn't exist.
I had no idea that I had 100 supplementals. I looked at each type and adressed the problem. I used this problem as an opportunity to clean house. Jim over at the Apache are is GREAT. If you look back through the history of that thread, you will see I'm not blowing smoke. I asked and solved a lot of problems about 2 months ago.
Finally, have faith. I know it's tough when you're losing traffic, but keep working at it. Matt tells us to clean up the supplemental (I took his advice) and not to worry about them - as long as you've addressed the reason they are there in the first place.
Good luck to all. And put that nervous energy to work in cleaning up your websites.
What I have noticed is that in the BD datacenters, when I do a site: search, there is a dup copy with https:// and http:// both being in the sup.
As far as redirects there are some 301's, no 302's but all pages active or redirected are sup.
The cache dates of the sup pages are very old, about 6 months. Non BD datacenters, the cache date is normal.
By looking at my sites that all have the https version in the cache, it appears that the whole problem could be from a canonicalization issue. I have also seen people post about both the www and non-www versions in the cache. Its almost like google saw two versions of each page either https and http or www and non-www or both and just decided to drop everything.
Is this what everyone is seeing?
Is this what everyone is seeing?
Yes...that is the nature of this bug.
Google, though, is indexing like crazy. This month alone I have had 128,786 googlebot hits (this does not include Mediapartners). Maybe a major refresh of the index?
[edited by: Grinler at 2:37 pm (utc) on Mar. 10, 2006]
<<<What i dont understand is why some authority sites have been nuked by pushing pages into the supplemental index yet some have managed to bypass it! >>>
I have noticed something strange. My very large site got nuked as well. 50% loss of google traffic due to supplemental issuues. It has been up and running about a year now and it went all supplemental.
Now, I put up another new website in December of 2005 and it is rising through the ranks very quickly and getting properly indexed. No sandbox, its getting page rank quickly, etc...
I have a theory that a lot of newer site are not being effected by the supplemental issue because there is no old data to cache.
Does anyone have any newer sites that are doing well? May be its a good time to build some newer sites if you are planning it.
I have a few really strong backlinks but it's like google didn't even notice them so I'm nowhere good in the search results.. I'm hoping that will change after Big Daddy updates ...
I used a google site map and as soon as I did google did start to crawl it fast and it indexed fully in 3 months.
My big site I also used a google sitemap and google crawled it like crazy as well, it just went supplemental and showed old cached pages.
It would be nice if when someone posts that their site has recovered from this "Supplemental Hell" they would also post the IP of the DC where they observed this. According to Matt Cutts, Google has been rotating BD DC's in and out. Therefore it is possible that a BD DC has simply reverted back to pre-BD temporarily. We've seen this happen the last month or so.
I'm inclined to just wait until mid next week before doing anything on a site.
Here is some information about my site that has gone to supplemental hell.
1. Home page is fine, All other pages are supplemental
2. Supplemetal listing show pages that are very old, most with cache dates back to Jan 05.
3. Many of the supplementals are pages that no longer exist and serve a true 404.
4. Quite a few of the supplementals are old pages that we asked be removed via the Google Removal Tool just a couple of months ago. There were gone, now with BD they are back again. Removed 2 months ago, so well before the 6 month window is up.
5. We are an E-commerce focused site
6. We are an Adwords Advertiser (paused all campaigns yesterday because we can no longer afford them due to the supplemental club membership)
7. We have a redirect in place for non-www to www although we have never had any non-www pages showing up in the serps.
8. The only traffic we are getting from the natural rankings are from our homepage's organic listing. None of the supplemental listings are giving us even a single referral.
9. Looking through the supplementals listings for our site, current pages that should NOT be supplemental all show a page size that is approximately identical in size "42K" regardless of what the real page size is. They do vary quite a bit because some have much more content than others.
10. Our site has existed for over 5 years. We last did a major redesign in Feb. 2005
We have spent alot of time on diversity because of Googles continuing headaches. These effort have provided some dividends, however some consumers use Google to qualify merchants. Kind of a "If your not in Google organic rankings you not worthy" mentality. So the supplemental hell problem is having some negative effect on us gaining new customers.
Like others, we have recently paused all of our Adwords campaigns because without the addition of natural rankings, we can not justify the low ROI we receive. Low ROI on Adwords is a common place in our industry, you have to also have the natural rankings to justify the Adwords spend.
I hope others are examining their ROI on Adwords to determine if a purely Adwords presence is worth it. Because if it is not, you might be better off spending that money on other things or even just cutting your expenses to help you financial ride out the problem.
Some times, it is not a wise move to just throw money at countering a problem like this with the same organization that caused the problem in the first place.
what I have been hearing the datacenters are being reset back to fix this problem in the Bid Daddy roll out
Mind sharing where you heard this? Thats a big move on google's part to roll back and I honestly do not think they would do it. Hope they do though :)
Cant see that happening either.
As pgillman said I am a bit worried that people who are saying there site has returned from supplemental hell are just seeing a non-big daddy dc or a Big Daddy DC not showing Big Daddy results.
What(or where) in particular? Would love to hope :P
Like others, we have recently paused all of our Adwords campaigns because without the addition of natural rankings, we can not justify the low ROI we receive. Low ROI on Adwords is a common place in our industry, you have to also have the natural rankings to justify the Adwords spend.
We paused ours over a week ago and won't reactivate until this situation is resolved. We moved the ad $$ to our MSN and Y accounts.
At the moment (at least where I am in Canada) Google has "reverted" to the most recent index from mid-FEB but no doubt the BD index from hell will return later in the day.
I think as tedster said a month or more ago, they tested BD on an index from about last August and planned to combine the BD algo with the current (FEB) index. However something unanticipated happened in how the old BD index and the FEB index merged. Its like all those old pages in BD triggered some sort of global filter which made them go supplemental, maybe because of age (as they were already 6 months old by definition in BD) or maybe even because appearing in both the old and new indexes they triggered the dup content filter.
Would love to hear them post that all is fine and I will gladly take back this post.
Cannonicals fixed in a week? I would be ecstatic if they were fixed this spring.
My newer site has just went up in the serps today. So there are sites that are benefiting from this mess.
Also I am reading that a lot of people are stopping adwords because they can no longer afford it due to their serps dropping. Does that mean that sales were coming from serps only and adwords clicks were not paying off? It seems that if you were using adwords to get to the number 1 spot on the sponsored search people should be clicking through and purchasing.
Any common factors found today may not apply next week....
I can very easily see the difference between the FEB index and BD as a have a site that I totally transformed between August when BD took its index and FEB.
BD is using August links and August PR. The switch is obvious as it goes back and forth from different datacenters if your site has changed in PR or substantially in links or pages from AUG to FEB.
What seems to be the problem is Google can't merge the BD index with the FEB index (or they are gradually merging it bit by bit). Every BD "merge" contains ancient results and huge numbers of pages defined as supplemental because of some error in reconciling the two indexes.
What is ugly is that they are doing it live, not caring how many sites lose mega money in the process.
Also I am reading that a lot of people are stopping adwords because they can no longer afford it due to their serps dropping. Does that mean that sales were coming from serps only and adwords clicks were not paying off? It seems that if you were using adwords to get to the number 1 spot on the sponsored search people should be clicking through and purchasing.
Shifting ad $$ to MSN & Y is the *only* way I have to protest.
