If google itself lost its traffic overnight im sure it would be acting a lot quicker!
Meanwhile whats the bl@@dy point? - have the serps improved - NO, junk is rising to the top and having a field day as a result
Are the serps more relevent - NO junk ranking well as a result of this major blunder
Did it need fixing? - NO - should have left it well alone in the first dam place!
Whats this achieved - Nothing but worse serps and the stuffing of a number of quality authority sites that have supported google pooring millions into adwords!
Do we deserve better than this? - YES
|The problem I am hearing for affected domains is different: urls that are still actively being served are being "eaten away" from the main indexes and into the supplemental index -- until there's no url left on a site: search that ISN'T supplemental but the Home Page. As supplemental, these pages will rarely rank for anything except oddball searches -- even though they were ranking beautifully in the very recent past. |
I dont think this is exactly what is happening. It is not that your actively served pages are being "eaten away" from the main indexes and into the supplemental index, but rather that these pages are being dropped from the index altogether. Eventually you are left with only your home page as a legitimate entry and then the rest being supp pages. At this point I can not find a single active url from my older pages (that are still visible here: 22.214.171.124 in the new index. They simply have been dropped and all I am left with is homepage and supps. From what I understand..that is the bug
All these pages though, to this day, are still being actively crawled by google.
Asher 02 said "The only missing thing for me is that I don't know if there are any other sites that were effected that did not use 301/302"
I have four site first one to suppleMENTAL hell had no 301 then today site with 301 did the same so im not sure it makes any difference.
Ok, perhaps a dumb question since I do not understand the inner workings of the G db. Supplementals here have perhaps a dozen old cache dates, one going back to Dec, 2004. Main page with recent cache. Question: why are there so many sets of old cache dates in the G db? Why not db containing sets of last few?
>> Question: why are there so many sets of old cache dates in the G db? Why not db containing sets of last few?
For each page in the index, there should only be one entry or else Google wouldn't know which page to serve. If you're seeing old cache dates, that because that is the only page in that particular index. I have some theories why Google holds onto pages forever, but that is just my guess.
My suggestions are these (this is what I did):
Look at the supplementals and figure out how they got there in the first place. The examples from my website and the examples folks sent me are all spawned from "sloppy" webmastering. I didn't 410 a page when I should have or I switched naming conventions and didn't redirect right away to the new page. I didn't block new pages from spawning using robots.txt. I didn't return a 404 when a page didn't exist.
I had no idea that I had 100 supplementals. I looked at each type and adressed the problem. I used this problem as an opportunity to clean house. Jim over at the Apache are is GREAT. If you look back through the history of that thread, you will see I'm not blowing smoke. I asked and solved a lot of problems about 2 months ago.
Finally, have faith. I know it's tough when you're losing traffic, but keep working at it. Matt tells us to clean up the supplemental (I took his advice) and not to worry about them - as long as you've addressed the reason they are there in the first place.
Good luck to all. And put that nervous energy to work in cleaning up your websites.
Same thing here:
Homepage is fine, and ranks decently. Every other page is supp.
What I have noticed is that in the BD datacenters, when I do a site: search, there is a dup copy with https:// and http:// both being in the sup.
As far as redirects there are some 301's, no 302's but all pages active or redirected are sup.
The cache dates of the sup pages are very old, about 6 months. Non BD datacenters, the cache date is normal.
By looking at my sites that all have the https version in the cache, it appears that the whole problem could be from a canonicalization issue. I have also seen people post about both the www and non-www versions in the cache. Its almost like google saw two versions of each page either https and http or www and non-www or both and just decided to drop everything.
I don't know about the rest of you, but My problem isn't actually that my pages all went to supplements, as I think the pages in the supplements deserve to be there. The problem is that the supplements show up at the top of my listings since every other page is gone.
Is this what everyone is seeing?
At my site (big shopping portal) the Mozilla-Bot is strongly active for months. The Mozilla-Bot had, however, a break from approx. 15 hours yesterday afternoon. After this the Bot started to index my pages again. It started at the initial page and proceeds structuredly -> a new Deepcrawl? Perhaps a positive sign (-> Re-indexing)? Watches somebody similar?
|Is this what everyone is seeing? |
Yes...that is the nature of this bug.
Google, though, is indexing like crazy. This month alone I have had 128,786 googlebot hits (this does not include Mediapartners). Maybe a major refresh of the index?
[edited by: Grinler at 2:37 pm (utc) on Mar. 10, 2006]
<<<What i dont understand is why some authority sites have been nuked by pushing pages into the supplemental index yet some have managed to bypass it! >>>
I have noticed something strange. My very large site got nuked as well. 50% loss of google traffic due to supplemental issuues. It has been up and running about a year now and it went all supplemental.
Now, I put up another new website in December of 2005 and it is rising through the ranks very quickly and getting properly indexed. No sandbox, its getting page rank quickly, etc...
I have a theory that a lot of newer site are not being effected by the supplemental issue because there is no old data to cache.
Does anyone have any newer sites that are doing well? May be its a good time to build some newer sites if you are planning it.
Well I have a fairly new site (about 3 months now) and only thing I see is that I actually don't see anything ;)
I have a few really strong backlinks but it's like google didn't even notice them so I'm nowhere good in the search results.. I'm hoping that will change after Big Daddy updates ...
I used a google site map and as soon as I did google did start to crawl it fast and it indexed fully in 3 months.
My big site I also used a google sitemap and google crawled it like crazy as well, it just went supplemental and showed old cached pages.
It's not that it isn't indexed by Google, it's just that Google isn't indexing the pages which point back at me, and I'm sure it would improve my position in the searches.. Oh well.
I just did a site search and am not supplemental anymore it looks like what I have been hearing the datacenters are being reset back to fix this problem in the Bid Daddy roll out. I assume Google will find out the cause fix it and redo the roll out again. I have also noticed the serps changing as pages that had dissapeared are now back in the search. Wonderful news just thought I would share it to all
I agree with BillyS. We've been observing some of these supplemental issues since May of last year and our supplementals were mostly canonical related (www versus non www, malformed URLs, and tracking code duplicate entries). We put in place 301s per Google's instructions and tried cleaning these up ourselves. Since mid last year, we've written Google and did get a response in January that they were working on a permanent fix to these canonical items. This same group also confirmed exactly what GG said - a fix should be in place and we should see sites re-indexed by mid next week.
It would be nice if when someone posts that their site has recovered from this "Supplemental Hell" they would also post the IP of the DC where they observed this. According to Matt Cutts, Google has been rotating BD DC's in and out. Therefore it is possible that a BD DC has simply reverted back to pre-BD temporarily. We've seen this happen the last month or so.
I'm inclined to just wait until mid next week before doing anything on a site.
I have been watching this thread from the beginning hoping wewould all find something or things we have in common, but it appears that although we may, there are alot of uncommon things floating about.
Here is some information about my site that has gone to supplemental hell.
1. Home page is fine, All other pages are supplemental
2. Supplemetal listing show pages that are very old, most with cache dates back to Jan 05.
3. Many of the supplementals are pages that no longer exist and serve a true 404.
4. Quite a few of the supplementals are old pages that we asked be removed via the Google Removal Tool just a couple of months ago. There were gone, now with BD they are back again. Removed 2 months ago, so well before the 6 month window is up.
5. We are an E-commerce focused site
6. We are an Adwords Advertiser (paused all campaigns yesterday because we can no longer afford them due to the supplemental club membership)
7. We have a redirect in place for non-www to www although we have never had any non-www pages showing up in the serps.
8. The only traffic we are getting from the natural rankings are from our homepage's organic listing. None of the supplemental listings are giving us even a single referral.
9. Looking through the supplementals listings for our site, current pages that should NOT be supplemental all show a page size that is approximately identical in size "42K" regardless of what the real page size is. They do vary quite a bit because some have much more content than others.
10. Our site has existed for over 5 years. We last did a major redesign in Feb. 2005
We have spent alot of time on diversity because of Googles continuing headaches. These effort have provided some dividends, however some consumers use Google to qualify merchants. Kind of a "If your not in Google organic rankings you not worthy" mentality. So the supplemental hell problem is having some negative effect on us gaining new customers.
Like others, we have recently paused all of our Adwords campaigns because without the addition of natural rankings, we can not justify the low ROI we receive. Low ROI on Adwords is a common place in our industry, you have to also have the natural rankings to justify the Adwords spend.
I hope others are examining their ROI on Adwords to determine if a purely Adwords presence is worth it. Because if it is not, you might be better off spending that money on other things or even just cutting your expenses to help you financial ride out the problem.
Some times, it is not a wise move to just throw money at countering a problem like this with the same organization that caused the problem in the first place.
|what I have been hearing the datacenters are being reset back to fix this problem in the Bid Daddy roll out |
Mind sharing where you heard this? Thats a big move on google's part to roll back and I honestly do not think they would do it. Hope they do though :)
>>>Thats a big move on google's part to roll back and I honestly do not think they would do it.
Cant see that happening either.
As pgillman said I am a bit worried that people who are saying there site has returned from supplemental hell are just seeing a non-big daddy dc or a Big Daddy DC not showing Big Daddy results.
a lot of what im seeing smacks of a rollback
"a lot of what im seeing smacks of a rollback "
What(or where) in particular? Would love to hope :P
|Like others, we have recently paused all of our Adwords campaigns because without the addition of natural rankings, we can not justify the low ROI we receive. Low ROI on Adwords is a common place in our industry, you have to also have the natural rankings to justify the Adwords spend. |
We paused ours over a week ago and won't reactivate until this situation is resolved. We moved the ad $$ to our MSN and Y accounts.
Same here i have paused about $1500 in adwords campaigns until things are straightened out. Using Overture for now on a lower budget.
I think trinorthlighting was correct that older sites are more affected.
At the moment (at least where I am in Canada) Google has "reverted" to the most recent index from mid-FEB but no doubt the BD index from hell will return later in the day.
I think as tedster said a month or more ago, they tested BD on an index from about last August and planned to combine the BD algo with the current (FEB) index. However something unanticipated happened in how the old BD index and the FEB index merged. Its like all those old pages in BD triggered some sort of global filter which made them go supplemental, maybe because of age (as they were already 6 months old by definition in BD) or maybe even because appearing in both the old and new indexes they triggered the dup content filter.
I personally don't think any aspect of this move is going well from Google's standpoint. If they were, Matt Cutts or GoogleGuy would be posting how things are on track and looking good. Notice how they have not been posting much.
Would love to hear them post that all is fine and I will gladly take back this post.
Cannonicals fixed in a week? I would be ecstatic if they were fixed this spring.
since we are for the most part at a loss, lets explore some different angles. Since G is so big on links=relavancy.
To anyone the meets Tedster's description of the supplemental doghouse...
Has your number of recognized backlinks changed all that much in the BD DC's?
What about PR for the supplemental pages? Home page?
Do you use relative or absolute links within your site?
Involvment in any link resources (farms, etc)? Just yes or no.
I re-read my post and wanted to clarify the canonical issue. We wrote to Google 2 weeks ago regarding this exact "supplemental hell" situation that is the topic of this thread and they confirmed what GG stated about this issue. First they were aware of the "supplemental" issue and lost internal pages and second, they were working on it and that we should see a fix next week. We've heard nothing regarding a canonical fix other than that was one area that BD was supposed to address, according to Matt Cutts.
Regarding GG, he posted here that we should see a re-index by next week. I'm not sure what else to expect - hourly updates? I haven't checked yet this morning, but Matt Cutts has not responded to any of the numerous postings on his blog. Others have pointed to this forum on his blog and MC did acknowledge how to register here.
I just wonder how all these thousands of pages will come out of the supplemental index. I have never had a page come out of googles supplemental index.
My newer site has just went up in the serps today. So there are sites that are benefiting from this mess.
Also I am reading that a lot of people are stopping adwords because they can no longer afford it due to their serps dropping. Does that mean that sales were coming from serps only and adwords clicks were not paying off? It seems that if you were using adwords to get to the number 1 spot on the sponsored search people should be clicking through and purchasing.
Seems like we need to wait till next week before wasting any more effort on this mess.
Any common factors found today may not apply next week....
I can very easily see the difference between the FEB index and BD as a have a site that I totally transformed between August when BD took its index and FEB.
BD is using August links and August PR. The switch is obvious as it goes back and forth from different datacenters if your site has changed in PR or substantially in links or pages from AUG to FEB.
What seems to be the problem is Google can't merge the BD index with the FEB index (or they are gradually merging it bit by bit). Every BD "merge" contains ancient results and huge numbers of pages defined as supplemental because of some error in reconciling the two indexes.
What is ugly is that they are doing it live, not caring how many sites lose mega money in the process.
|Also I am reading that a lot of people are stopping adwords because they can no longer afford it due to their serps dropping. Does that mean that sales were coming from serps only and adwords clicks were not paying off? It seems that if you were using adwords to get to the number 1 spot on the sponsored search people should be clicking through and purchasing. |
Shifting ad $$ to MSN & Y is the *only* way I have to protest.
pgillman, thanks for the clarification. It is great you got a response from Google. I have written them before and they never had the courtesy to respond. Yes, I do think even the big corporations need to be courteous.
|BD is using August links and August PR. The switch is obvious as it goes back and forth from different datacenters if your site has changed in PR or substantially in links or pages from AUG to FEB. |
bobmark, I find that every interesting and I hope that explains this mess. I have a site that was launched in November. I did well (picking up sandbox type traffic) until February 1st. Then it tanked. Recovered just a little in the last few days. If what you are saying is true, that would explain what I am experiencing with that site. Anyone else launch a new site between August and February. I must say that none of my pages are supplemental though, just talking about traffic here.
| This 210 message thread spans 7 pages: < < 210 ( 1 2  4 5 6 7 ) > > |