Loss of indexed pages

[keyplyr]

Over the last 10 days I have seen a 20% gradual reduction of indexed pages shown in GWT > Crawl > Sitemaps.

GWT > Crawl > Sitemap: shows no errors.
GWT > Crawl > Crawl Errors: shows no errors.

Manually looking through raw access logs shows no occurrence of Googlebot blocked from authenticated crawl ranges and I do not serve different content to different countries/languages. Site is 20 years old, hard coded & responsive-mobile (no errors.) My traffic remains high and seemingly unaffected by this.

Could this be Google reassessing what content gets indexed, possibly in relation to April SERP changes? Anyone else seeing anything similar?

[Atomic]

Yes, the exact same thing has been happening to one of my sites. WMT indicates fewer indexed pages, but I hand checked them and they are showing up for searches and still appear when I do a site:domain search. At first I was very worried, now just curious.

[keyplyr]

How did you determine which pages were supposedly un-indexed?

[Atomic]

I have lots of sitemaps for the purpose of being able to tell which pages are or are not indexed. I picked several sections with supposedly un-indexed pages and searched using text from each page until every page had been tested. All turned up in the SERPs. So, I don't know which were supposedly un-indexed. I just know that none are.

[Wilburforce]

WMT shows a ~15% drop in the number of indexed pages for my site between 29 March and today.

I haven't added or removed pages (or resubmitted the sitemap) during that time, and can't see any obvious reason for it, but I tried clicking Update on the Advanced tab and ten pages came back, so I don't think it is anything to lose sleep over.

[lucy24]

Could this be Google reassessing what content gets indexed

Honestly I think it's google sweeping behind the WMT computers and a few things getting temporarily unplugged. It always gets back to normal within a week or two.

[Itanium]

Same here. Indexed sitemap pages dropped by 30%, but the SERPs don't seem to be affected.

[keyplyr]

but I tried clicking Update on the Advanced tab and ten pages came back,

I'm not referring to Index Status numbers, which always seem to be in their own world. As mentioned in the OP, I am referring to the total number of indexed pages in Sitemaps.

[Wilburforce]

the total number of indexed pages in Sitemaps

That was showing the same drop over the same period until I updated the index status, but the number indexed hasn't been exactly the same as the number submitted - although usually within a smaller margin - for as long as I can remember.

I'm with lucy24 on this one.

[keyplyr]

although usually within a smaller margin

My experience is the indexed number is usually 100% of my sitemap pages, dropping a page or two or three every so often, then returning. I have never seen it drop 40 pages as it has recently.

First it dropped 8 pages, then down to 14... and so on over a couple weeks to now 40 missing pages so this does not appear to be a glitch. My site has too many pages to count manually, so I don't know which have been removed from the imdex. Too bad GWT doesn't tell us.

[keyplyr]

UPDATE

GWT showing return of a dozen pages to the index. Hopefully this trend will continue.

[Wilburforce]

GWT showing return of a dozen pages to the index.

Not in my case: I lost another page this morning. I slept well, however.

[Kratos]

@keyplyr If it helps I have heard of other webmasters getting their number of indexed pages drop in the last weeks. We have a group of webmasters who chat on Skype and one of the topics was that there was a drop of up to 5% of indexed pages. Only a couple of the guys reported it and we came to the conclusion that it's a glitch as seeing how buggy webmaster tools has been for the last 4 weeks.

They're probably getting flooded with webmaster tools requests seeing how close we are tot he 21st and how webmaster tools has become so buggy over the last 4 weeks. All kinds of glitches in reporting.

[keyplyr]

I do not believe what we're seeing is a glitch. It's too incremental. Could be testing of new algo.

[Itanium]

@keyplyr same here. 400 pages back in the sitemap-index.

[samwest]

maybe they are considering them doorway pages...if you think they were not doorway pages, google thinks differently and is notorious for false positives. more fun from the planet's content judge & jury.

[keyplyr]

No doorway pages.

Still 2 dozen pages less that what should be indexed, and still showing no errors anywhere in GWT.

[samwest]

It doesn't matter how many pages I include in my sitemaps, G always restricts about 10% and will never index all pages. I see that on some of my client sites too.

[keyplyr]

Not the case for my sites. All pages I include in sitemap.xml are indexed. Low quality, low content pages are left out of sitemap.xml as they should be. For GWT to not index ALL the pages I do include is abnormal for me, hense the thread.

[lucy24]

It doesn't matter how many pages I include in my sitemaps, G always restricts about 10% and will never index all pages.

Don't believe everything you read in wmt. My personal site has two visible directories, so the sitemap lists exactly three pages-- that is, the three directory-index pages, because google can jolly well figure out the rest for itself. As we speak, wmt says there are three (3) pages submitted, of which two (2) are indexed.

A quick detour to Google Search with a site: operator tells me that they have miscounted.

[keyplyr]

The downward trend continued to where only a third of the total pages were showing as indexed, very disturbing. I posted in Google Groups and got several odd replies that made no sense :)

Then today, all of a sudden, every page showed as indexed. Well... whadoya know!

[samwest]

I noticed a phenomenon on a recent page trimming exercise...before I started I had (for example) 200 pages. Only about 160 of those pages were indexed.
Of course there is no really easy way to determine what those pages are...by design I suppose.

Once I trimmed that down to HALF those pages (for example only, don't be quoting me this in a future post cause I'm not longer sharing exact numbers) that ratio stayed exactly the same, only 80 pages were indexed.

If you're still shedding pages, it's just more of the Google serps slaughter. They appear to have done it again on May 1st.

[keyplyr]

@samwest - Thanks, but read my last post. It was a Google bug after all.

[Broadway]

I noticed this problem about a month ago and posted about it but somehow overlooked this thread here.

I have a 200 page site. Over the last month, WMT, both on the Sitemaps page and on the Index Status page, has shown wide fluctuations in the number of pages indexed. Also, the number shown on each of these pages never agrees.

Right now the Sitemaps page shows 198 pages indexed whereas the Index Status page shows only 131 (although, admittedly that number is dated last week).

When I initially noticed the problem, I "site:example.com" searched for all of my pages (I took the url's directly out of my sitemap). Every page was in the index, despite what WMT reported.

80% of the traffic to this site is through search. My traffic never suffered at all, yet at times WMT stated that only about 1/2 of my pages were indexed. There is no way I could have that many pages not in the index and not suffer a traffic loss.

WMT should remove those reporting tools until they are ready for prime time, because evidently they have absolutely no validity. The woman down the street reading Tarot cards has just as much credibility as WMT on this point.

[keyplyr]

+1

[seoskunk]

I see it more as a rollback certainly the amount of indexed pages have fluctuated in last week but I am wondering if this is just a roll back in preparation for new panda update.
I see the percentage of indexed pages the same as pre panda.

[keyplyr]

This is definitely a bug in WMT reports which may or may not have been fixed for everyone.

[samwest]

So after you realized it's a bug, how is your actual real world traffic doing?

[keyplyr]

Traffic was not affected by the inaccuracies of the report. But, because of Google's mobile search update & a few other factors, my traffic has about doubled.