No of Indexed Pages Decreased by 8%

[vlexo]

Hey all,

Looking to get some help here. Looking in GWT/Search Console the site I work on has seen an 8% decrease in the amount of pages indexed, which equates to around 10k pages.

I've narrowed this down to our forum, which has almost 200k pages (threads, posts, userids pages etc).

I can't find an example of a page that has been de-indexed, because unfortunately that's like going through a needle in a haystack. Unless there's an easier way of identifying which pages Google aren't indexing anymore?

Anyhow, the amount of pages that have been de-indexed seems to have occurred over May.

The forum hasn't changed (with perhaps people posting being the only change), so what are the reasons, essentially, why Google would choose to de-index 10k forum pages? Is this normal for a forum? Does this just mean that that those 10k pages on the forum have lost their relevancy over time? Or perhaps people were posting the same sort of content and Google have picked a preferred thread to rank/index?

[Broadway]

Recently I had posted that WMT reported a drop in indexed pages for my 200 page site. At one point it stated about half were deindexed.

I "hand" verified that all 200 were in fact still in the index.
Further proof was that website traffic never faultered (this site gets pretty much all of its traffic from search).

Now a month later, all of the WMT reporting has returned to normal.

You may not really have a problem.

[vlexo]

Hi Broadway, thanks for that nugget of info. It's strange why Google would provide inaccurate data on that level. Any else face same 'issue'?

[msenza]

I've also observed this on 3 the sites I manage... One almost a complete unindex, keeping 10% of the page, the others around 20% also...

[Johan007]

It would be very handy if in WMT you could view all your indexed pages with options to filter. The "site" query is not good enough. I don't see why Google would not do this?

[lucy24]

It would be very handy if in WMT

... where it shows the number of indexed pages, you could view or download a listing of exactly which pages they are for any given date. ("Or download" because I guess viewing is not really practical if you have a million pages :)) I was looking up something else in wmt and noticed that on some recent date the total number of indexed pages dropped by two. Now, I know I haven't deleted two pages recently (I do not have a million-page site), so are they talking about pages that I 410'd months ago and they've only now got around to removing from the index? Or was it just google math, like when you do a search and it says "4 results" and the SERP lists 17 pages?

Always remember: Information given in Webmaster Tools (everyone's, not just google) isn't about what's happening in Google. It's about what's happening in WMT. So 99 times out of 100 the best immediate response is to do nothing, sit back and wait for the problem to sort itself out.

[Paperchaser]

Yes i do see the same pattern as well at first i didnt really payed attention but ill keep an eye on it.

[Johan007]

@lucy24

where it shows the number of indexed pages, you could view or download a listing of exactly which pages they are for any given date.

This downloads the total pages indexed for a given date with no URL's?

[lucy24]

I'm saying it would be nice if such a thing existed. Currently it doesn't :( Yes, I meant specific URLs, not just aggregate numbers. It's no use saying that last week 995 pages were indexed and this week it's 987, if you can't look at a list and find the differences. Right now you don't even know if it's simply 8 pages gone ... or 45 gone and 37 added. (I'm making up the numbers.) GWT actually says "pages" ... but it really should be "URLs" shouldn't it?

[aristotle]

If the index has gradually gotten bloated and big and unwieldy, it might make sense to clean out worthless pages that hardly get any traffic anyway.

[lucy24]

it might make sense to clean out worthless pages that hardly get any traffic anyway

Well, it would definitely make sense for a site owner to do that kind of housekeeping. But that's not how Google works, is it? For as long as I can remember, their goal has been to index everything, everywhere. Raw logs, material in languages they don't know, you name it. (Much like that perennial caution given to parents of toddlers, beginning with "If they can see it" and ending with "they can choke on it". If you build it they will index it.) Suppose someone searches for an exact phrase that only occurs on one page in the world? There's a place for search engines with that kind of data at their fingertips.

[aristotle]

Well Lucy -- imagine that a billion chimps, each with its own keyboard, continuously type letters more or less at random, and every 50 lines that each chimp types is automatically turned into a web page. Just by luck, every once in a great while, one of them types out a beautiful poem, but everything else is garbled garbage. In your opinion, should Google include all of those pages in its index, just in case 20 years from now someone searches for an exact phrase in quotes that matches a string of garbled garbage from one of those pages.

[lucy24]

should Google include all of those pages

Not necessarily Google, but someone should. And not necessarily as their only option: there could be "good" pages and "not-so-good" pages.

Or consider this, picked at random from an actual site (not my own):

tr5gk5 x3F3ix6tk5, xJE6hwpk5, isF6tk5, X`oyk5, wo8ix6ttpk5, cspmp3Jxk5, x7ml c9l`N31]Z6gk5 Z?moEpk5

Garbage, right? Raw OCR from a device set to the wrong script? The latest from the webspam [google.com] site?

Wrong. It's simply in a legacy font. Google doesn't know this-- but the potential searcher does. That's why someone, somewhere, has to index everything.

[aristotle]

I can see your point of view, Lucy. But I also think that practical difficulties will eventually intervene to limit what can be done.