Forum Moderators: Robert Charlton & goodroi
[webmasterworld.com...]
and here
[webmasterworld.com...]
A lot of members are seeing huge sites going supplemental. One of our main sites lots all rankings and 200,000 + pages disappeared and now we are left with 19k useless results. This could be a goof or it could be a new round of penalties. If you have had your site reduced to the 'sup index lets here about it and compare notes.
I was going through my supplemental site now, did a site:www.mydoamin.com and was looking for clues as to the reason and dubus head saw and mydomain.com/products/aspid=222
Shoot look at your site hard it was my fault Google is correct I just hope it will correct itself as I deleted all the files now so it is dead pages.
It is reassuring to know we are not the only ones this has happened to, and all theories, however crackpot they might seem, are welcome because the more we all chip in the more likely it is someone will have that Eureka moment and the bits will fall into placeOK, put on your tin-foil hats, gather up, and listen carefully. I'm only gonna say this once! ;)
The key thing to remember is that BigDaddy is being deployed to resolve canonical problems.
I suspect sites are going supplemental down to the home page to identify the canonical domain. Then the site is respidered and rebuilt in Google's index using "new infrastructure," revised code that better handles canonical issues.
Many here have reported:
1. A page count drop and/or pages going supplemental.
2. Until the home page is the only page indexed.
3. Heavy spidering.
4. Pages coming back.
5. Site fully indexed.
Google has 20-25 billion pages indexed so this sequence is gonna take some time to repeat on all of them. They can only process "a few" sites per day. Perhaps one percent? And they're doing this on "live" DCs because it's the only place they have enough storage.
I don't follow DCs or try to monitor BigDaddy but I saw traffic tank and then resume on two well established white hat sites in January. The full cycle took about three to four weeks and the sites are now back, fully indexed, and doing as well as ever. One uses a Google site map, the other doesn't.
I'm uncertain about how Google decided which sites to process first or decides which to process next. Any ideas?
Bottom line, when the BigDaddy update hits your site, just sit tight and ride it out. And PLEASE, try not to whine too much, OK? :)
Good post my man. Could well be.
Background - BD HP Good, rest supplemental, default google All Good,
I noticed a supplemental result that looked like [mydomain.com...]
clicked on that it resolved ok but without style sheets, clicking on the menu links I got
[mydomain.com...]
[mydomain.com...]
[mydomain.com...]
All resolving to real pages but with // in the url instead of a single slash.
Looked deep and a lot not all of the supplements are // based
Looking at the site live with an editor and searching I did not find any url or links with a mysite.com//
Any Thoughts on how to get rid of those refferences in the index and why they point to resolving pages.
If you are curious or think you can help PM for the url.
Thanks
.
Three useful searches:
site:domain.com
site:domain.com -inurl:www
site:www.domain.com
ONE of those should return ZERO results.
[edited by: g1smd at 11:24 pm (utc) on Mar. 3, 2006]
How?
I am on a windows shared server, I have done asp 301 redirects on individual pages for non www and other issues, but there I had a real page on the site to add the code to. Here I do not have a page with mydomain.com//page.htm so where do I put the redirect?
A few people have had this, and it may be that someone is trying break your site apart with a link pointing to .com\\
However given what DaveAtIFG has just said you might want to wait and see if the canonical issue is resolved by Google for your site.
oof that was quick g1smd!
301...I'd look at the global asa...but it might be easier to wait or set up base hrefs depending on the number pages/urls affected
[edited by: tantalus at 11:36 pm (utc) on Mar. 3, 2006]
Yes you do, I thought that you said that the URL serves content. (If it does not serve content, and simply gives a 404 then ignore it, it will go away on its own soon enough).
So, add a detector and a redirect on the page that says "If we are at the '//' URL then 301 redirect to the '/' URL".
Of course, I should have guessed this was some sort of IIS screwup; you can also use ISAPI_rewrite to fix this (but there is a 99% chance that your host will have no idea what you are talking about, and a 99.9% chance that they won't let you use ISAPI or have any other sort of fix installed [Hint: next time, use an Apache webserver if you can]).
[edited by: g1smd at 11:32 pm (utc) on Mar. 3, 2006]
All I can say is
STEP AWAY FROM THE COMPUTER.
Go talk to some real people. Have a beer. Come back Monday.
You can't do anything about it anyway. Strange things have been happening for two years - white hat sites dropping 100 places and then coming back a month later, sites not even found for their own unique names, 302s, Canonical problems, duplicate content filters.
Google will do what Google will do. I hope they sort out all the glitches. At least big changes mean that something is happening - they have been in a mess for a long time now.
However, we have only our index pages listed and we are not seeing any spidering today. Time to be concerned?
Yep and this is one of the reasons we removed the google search box.
This search on 66.249.93.104
site:www.res ource-zone.com -inurl:showthread.php -inurl:forumdisplay.php
gave a few "normal" results (as expected) followed by several hundred URL-only results (as expected - they are URLs excluded by robots.txt and are slowly being delisted),
BUT, when I clicked on the "Search for English results only." link at the top left of the page, I suddenly got 22 000 results and all except the very first three were marked as supplemental results. These results are all pages that are disallowed in robots.txt and in a normal search have already dropped out of the index.
As soon as you specify "English only" they all re-appear as Supplemental Results (proof yet again, that Google does not delete old data, but merely hides it from search results, except that numerous glitches bring it back in some searches when it should still remain hidden).
Something is in the works, and I have no idea what it is.
The &sa=N parameter also changed to &sa=X&oi=lrtip7 in the search URL too.
Good theory. We've ridden out storms before. Let's hope it's as smooth as you predict :-)
"...but why are you burning the Rum?"
Why would it still be shown as supplemental?
DaveAtIFG, very sound reasoning...I'm very positive that your scientific prediction would turn out to be true!
Thank you for giving us an hope!
Do you mean that after the storm a site should regain all the pages instead the only supplemental ones in an automatic way?
Or do we need to clean all the "lost" page in our hosting? Do we need a good cleaning?
BTW - While all this is going on one of my competitors is running riot with 4 mirrors, traffic going through the roof and no one in sight to stop them.
First up - No traffic loss here.
But, I have a problem with this theory, well actually not with the theory, but with the Google execution!
Why make all these pages sup for this reason?
Why not simply leave the results "as is" and then respider from the known root.....Google knows the root and could remove pages accordingly!
This is a bit like the guy that says he has a water leak somewhere in his house, and someone says the solutuion is to demolish the house and rebuild brick by brick to eliminate the fault!
I'm not saying that Google isn't doing this, just that it appears like a dumb way to fix the problem.
Remember, he said he will only say it once. He is a man of few words and If I know Dave as well as I think I do, he will not post again. ;)
Personally I am not sure about that theory - we will see.
>>>>>Hi DaveAtIFG! then this would be happening to many more sites actually. Which would mean that all sites including WebmasterWorld would have to go supplemental until it was reindexed there.
WebmasterWorld has not got a canonical problem so this would not have required a de-indexing and then possible re-indexing. Google should easily be able to tell which sites have canonical issues - a reindexing would be a very very good thing for these sites.
We will see.
>> Why would it still be shown as supplemental? <<
As long the www pages are indexed OK, don't worry too much about some non-www pages still showing as Supplemetal. It takes Google one to two years to drop them from the index. It is a bug.
When you do a site:www.domain.com search you want to make sure that you do not hit the "In order to show you the most relevant results, we have omitted some entries very similar to the 2 already displayed. If you like, you can repeat the search with the omitted results included." message until most of your pages have been listed.
If that message comes up after only a few pages, say "1 to 5 of about 50" or "1 to 80 of about 800", then make sure that all of your titles, meta descriptions and page content are vastly different.
A "short" listing means the pages are filtered as being duplicate content. Having the same title and/or meta description on multiple pages is enough to trigger it.
Canonical problems are being sorted hold on to your hats and ride the storm.
If you read Post #400 in this other thread [webmasterworld.com] I think your questions will be answered. :-)
Why make all these pages sup for this reason?
You are not checking your results properly, pages which deserve to be supplemental as judged by google algorithm are now showing as supplemental results.
Legitimate pages are the ones that dropped out of the index go check again. Only legitimate page google now has for affected sites is your homepage. Take Dave's words he made the most appropriate guess. Nothing to panic just to wait this out.
Only ironic thing google should have tested these things before making bid daddy DCs live. They had a lot of time to do that. They shouldn't have affected millions of sites and their businesses by making huge changes in live datacenters.
That's because Google now believes they are the web.
