Why are more and more of my pages going supplemental?

No onsite changes and more links!

[marvin]

I've been reading the posts here about supplemental pages and I have a good idea as to what causes urls to be put in to the supplemental index.

However, I don't understand why lots of my pages have been moved from the main index in to the supplemental index recently. About 2 months ago around 300 of my site's 440 pages were in the supplemental index. This has been slowly growing and now about 360 pages are in the supplemental index (with a corresponding reduction to the number of pages in the main index).

I find this strange as I have made only trivial changes to the pages on my site and I have received several new links over this period.

At this rate i'm getting concerned that my whole site will go supplemental before Christmas! I would welcome any suggestions as to why this is happening and how I can remedy the situation.

Regards

Marvin

[tedster]

There appears to be some kind of shift going on here - in previous years, the end of August has been a time of upheaval at Google Search. I recently did an analysis of a site and found severl very high PR pages (PR5 to PR7) in the Supplemental Index, and I'm still chewing on the reasons.

Has this change affected your Google traffic, marvin?

[glengara]

How are we distinguishing what's supplemental these days, omitted results?

[tedster]

I've been assuming that site:example.com/* gives only regular index results. Then site:example.com gives the number of all indexed urls, and the difference is the number of urls in the Supplemental index. site:example.com/& seems to give the Supplemental number, too, and the numbers add up all around.

But caution here, both these queries are hacks and their accuracy is potentially off. Still, they were in line with the results that Google gave back when the Supplemental Results tags were active.

The "Omitted Results" link was not ever an indicator of Supplemental urls, only of similar page filtering, just as the message says.

[Tonearm]

I get:

site:www.example.com - 670
site:www.example.com/* - 120
site:www.example.com/& - 630

So they aren't adding up over here.

[tedster]

I just tried one - you're right, it looks like the & hack has completely blown up. The * hack still feels like it's in the ball park to me, however.

Looks like Google really wants us to forget about Supplementals.

[Bones]

The site: query in general has been giving frequently changing results for the last week or two, so I think they've been playing with it somewhat lately.

[TheSeoDude]

Looks like Google really wants us to forget about Supplementals.

I think google considers us stupid if they imagine we can forget about supplementals when entire sites have sunk inside. The traffic vanishes too and it's rather hard not to notice the visitor drop even if you want to please the she-google!

[SEOPTI]

TheSeoDude, exactly what I think about Google.

[McMohan]

Well, I am also the one who has witnessed steady decrease in the number of pages in the main index, as per example.com/* hack.

Quite logically, either this hack is working properly or Google has set us on a wild goose chase on this hack. The only way to be sure that this hack is working properly is to see if a page that you think has gone into supplemental index in the last fortnight ranks for any reasonable keyword. If it does, then we can dismiss this hack.

[nomis5]

Figures for two UK sites almost add up:

site:www.example.co.uk - 759
site:www.example.co.uk/* - 437
site:www.example.co.uk/& - 317

site:www.example.co.uk - 153
site:www.example.co.uk/* - 86
site:www.example.co.uk/& - 69

It's been around just under half in supplemental for ages irrespective of the number of pages on the sites.

I just get the feeling that something is going on at Google that is absorbing a lot of computer power. Maybe they are placing more pages in supplemental to reduce the load or possibly in preparation for an anticipated increase in load. The move to supplemantal is not affecting everyone but it does seem to have affected a significant number.
Maybe asking if anyone has seen a decrease in supplemental results will add some information?

[nomis5]

"The only way to be sure that this hack is working properly is to see if a page that you think has gone into supplemental index in the last fortnight ranks for any reasonable keyword. If it does, then we can dismiss this hack."

Investigated a bit along the lines above. I have a page that appears in the site /& results. It has a PR of 2. When I search on the relevant keywords the page appears as number 3 in both google.co.uk and .com.
So one thing for sure, the /& results are misleading at the least.

[tedster]

I think we may be experiencing some of the promised "improvements" to the supplemental index. If so, forget anything you thought you knew about it.

[TheSeoDude]

The site:domain.com/& hack does not work.
But the site:domain.com and site domain.com/* (non-supps) works.

As the pages I see listed as non-supplemental are the ones bringing traffic.

[jeffposaka]

I think Google's data centers are getting full.

[Tonearm]

I think tedster is right. I think we're about to see big changes. I'm thinking very early September.

[europeforvisitors]

Figures for two UK sites almost add up

If I add up /* and /& results, I get almost exactly the same total (5140) as I do with a plain-vanilla site: search (5130). [Both of those numbers sound a bit low to me, but then, I haven't counted my pages lately.]

[Bewenched]

Our site traffice has been bouncing around for days.

One day we are so busy we cant keep up or put the phone down without it ringing again. Then a day or so later, it's slow.

I would think with a 10 year old site we'd be pretty stable. Thank goodness for the long tail keywords.

[tedster]

If I add up /* and /& results, I get almost exactly the same total

Good to hear, EFV. I'm hoping to see the same again, soon. Still broken for me. The & hack gives me the same results as a straight site:example.com query, at least on the few domains I've tried. My default connection goes to 64.233.169.104 right now.

[Tonearm]

Mine started adding up properly this morning.

[marvin]

Some of my pages have been creeping back in to the main index now.

I was getting one or two pages page per day since starting this thread then this morning I noticed that 20 pages were back in the main index.

Anybody else noticed anything similar?

[europeforvisitors]

Mine started adding up properly this morning.

Mine are adding up a bit worse than they were: 5,360 with the "site" command and 5,290 when /* and /& are added. That's a disparity of 70 compared to 10 the other day. I'm not going to spend any time agonizing over it, since (a) different data centers may return different totals at different times, and (b) I'm not convinced that it matters.

[g1smd]

Different datacentres always have a few URLs that are different in their respective indexes.

Over time, and I mean many months or even a year or more, sometimes the reason becomes apparent, but often it does not.

Xenu LinkSleuth is useful to make sure that all your internal linking is in order. I would check your site using that.

[SEOPTI]

What's interesting, Google does not need to apply a new cache date for a page in order to remove it from suppl. index. PR can flow througout the whole site without visible cache date changes. They also don't need to apply a new cache date for a page to put it into the suppl. index.

This tells me that cache dates change at the Plex, but we don't see it.

It's time to say that Google does not export cache dates regularly, this is similar as with exporting PR.

But who knows, I think this may change again when they stop compressing the suppl. index.

[edited by: SEOPTI at 8:40 pm (utc) on Sep. 2, 2007]

[g1smd]

Matt Cutts said recently that the cache date should show the last date of spidering.

Maybe it does, but with a time lag?

[SEOPTI]

Yes, I think there is a huge time lag, but not with all sites.

[Tonearm]

My site:www.example.com and site:www.example.com/& numbers are currently exactly the same. Still roughly the same number in site:www.example.com/*. Too bad, I was hoping to see a positive change by today.

[SEOPTI]

They stopped indexing again, again I see an increase in supplementals. It seems like they are having some major problems at the Plex.

These are the numbers for site:www.example.* for one domain. Huge fluctuations. It's interesting how they add and remove pages from supplemental index. Those are pages in the main index, it changes on a daily basis:

8230 - 12.300 - 14.600 - 16.100 - 17.800 - 19.100 - 21.900 - 23.100 - 25.100 - 26.100 - 27.100 - 29.100 - 29.400 - 29.800 - 30.200 - 29.500
- 28.800 - 27000 - 25900 - 23600 - 22700 - 21700 - 20600 - 18200 - 17000 - 14700 - 13600 - 13000 - 12400 - 13200 - 14000 - 14900 - 15800 - 16800 - 17700 - 19600 - 20600 - 21500 - 22500 - 23400 - 23600 - 23200

[edited by: SEOPTI at 1:06 pm (utc) on Sep. 5, 2007]

[g1smd]

Is that one figure per day, or what?

I assume you use the same datacentre every time too?

[SEOPTI]

No, this is over the course of 1-2 months. When you look at the last number which is today, you see a decrease. Datacenters don't play a role for me.

The most interesting part is, that the number of supplementals directly affects traffic numbers. When I do site:www.example.* and watch the number of supplementals increases, I can be 100% sure that the traffic decreases also. Supplementals are 100% directly related to traffic levels.

All other factors have been consistent during this test, site structure, inbound links, outbound links, nothing has been changed. It's clear that Google plays with their suppl. index all the time.

[edited by: SEOPTI at 1:45 pm (utc) on Sep. 5, 2007]