Google DROPPING pages from results completely

[c41lum]

Hi everyone I need a bit of advice.

I have noticed in the past couple of days (7 to be precise) that G has totally removed some of our very high ranking pages. These pages haven't dropped into the supplemental, they have been totally de-indexed. they can't be found by doing a unique "string search" or doing a 'site' command search. Iv checked all the usual reasons i.e duplication, 301 HiJack, unreachable URL, cloaking, malware, keyword stuffing ect.

The pages that have been removed seem to be totally random i.e Green Widgets (de-indexed) while Blue Widgets is ranking on the first page like it has always done. Both pages have the same inbound links.

Has anybody else notice G removing pages from the index completely? normally I have seen pages drop and in some cases drop in to supplemental but I have never seen a total de-index. These pages are about 7 years old and are updated weekly.

Is this a new thing?

[tedster]

There was a rash of this a couple months back - then gradually those pages returned. Sorry to hear that it's happening again. It sounds like some back end process at Google gets buggy, doesn't it.

You said these were high ranking pages - but of course the damage done also depends on the specific query term. Were they also high search traffic pages?

[edited by: tedster at 1:30 am (utc) on Dec 18, 2010]

[scottsonline]

We lost ten percent about a week or two ago. Started coming back slowly. Happens every few months. I'm betting google has no idea as wmt never changes but it 100% correlates to dropped traffic

[c41lum]

Hi Tedster, they were mid search traffic terms.

Whats blown my mind is the total disappearance, normally if we get a penalty we will drop -50 or -950, this is different these pages have removed, there also seems to be no pattern to which pages that have been removed. We still have some very good high search term positions. Traffic is down about -30%.

Has anyone else seen this lately? and if so did your site recover or did you change the URLS for the pages which is one option im contemplating.

[sandboxsam]

I am seeing many of my key internal pages disappearing from the index too. Also started about two weeks ago.

[Josuah1973]

I have a small site where I have suffer the same thing, some high ranking pages removed from the index, starting the monday 13th.

[HuskyPup]

I've had thousands of images removed in one fell swoop at the beginning of October and absolutely no idea why except that many of them were 10+ years old even though they are the accepted de facto trade widget standard images used by many globally.

Some have returned to the index, 90% have not, however I still rank mostly #1 for the regular SERPs...sorry, not a clue what they're fubarring these days!

I have to also say that other widget companies were also hammered at the same time and I see my images on formerly very low ranked sites at the top of the image results now.

[jmccormac]

There seems to be a vast discrepancy between the number of results returned on Google for a site type search and the number of pages listed as being in the index in WMT. It is also very disconcerting to see page counts dropping by the day.

Regards...jmcc

[cien]

I've lost close to 60K pages from the G's index in the last 8 days. Mayday all over again it appears.

[jmccormac]

@cien But are the 60K pages still listed in the Webmastertools url count? It could be some bozo attempt by Google at gauging a site's most authoritative pages based on the ones most frequently accessed. On deep sites,(ones with thousands or millions of pages), this won't work well because the entry point might be a unique url that could only be hit once (having been found in a search). The page count issue still seems to be in flux because searching on www. and on the domain name can give very different results.

Regards...jmcc

[c41lum]

Still no return of my pages. when i do a site: search google shows 251,000 pages, we only have 30,000! This is a bad sign and normally I would expect to find lots of duplicate pages when the number of pages jumps like this. In this case Iv not found any duplication.

Has anyone recovered from this? or is it time for me to move the content to a new URL on the same domain..

[cien]

@cien But are the 60K pages still listed in the Webmastertools url count? It could be some bozo attempt by Google at gauging a site's most authoritative pages based on the ones most frequently accessed.

@jmccormac, could be, not sure. WMT reports twice the amount of pages site:mysite.com reports. I don't pay much attention to WMT's indexed pages reporting. I've found WMT indexing reporting to be not reliable. My traffic always remained the same. So I don't trust it that much. However, when any fluctuations on the amount of pages on site:mysite.com occurs, it does affect traffic to my site like now.

[sandboxsam]

Saw many internal pages disappear from the G index on/or around Dec 7.
www.site.com/* has decrease by 41% over the past 10 days.
None of the missing pages have returned yet.

[c41lum]

@sandboxsam that's almost exactly what Im seeing. Mine was more like the 10th DEC though.

Non of mine have returned, not even in the supplemental. Its like they have been black holed. Pages that link from the de-indexed pages have seen a slight drop, not as big as I would have expected though.

I think we should try and see if there is a pattern to the disappearing pages. Here's what Iv noticed so far:

Pages that have disappeared/de-indexed are "brand" related pages not product pages or category pages. When I do a site: search the number returned is Very Very high x3 what it normally is plus the home page is now sitting on page 2 it was one page 1 second position prior to the 10th Dec. Traffic is down between 30% and 40%. These pages have not been optimized or aggressively linked to. Pages are about 7 years old. Fresh content is added on a daily/weekly basis.

Not all brand pages have disappeared it only seems like 30% of the brand pages have been de-indexed. Iv checked the usual suspects i.e duplicates, hi jacks, blocked by robots and on page no follow and every thing seems fine.

[netfleet]

Mate of mine has had something like this happen over the last few weeks but the site has been coming back then going again.

Homepage was ranking #1 for a high volume search term (801M results) then in a flash de-indexed. Then 2 days later right back to #1. Then yesterday (2 days later) completely gone again.

He does use some pretty aggressive and grey-hat SEO though.

[sandboxsam]

My dropped pages started coming back yesterday and look like most are back now, approximately 30 days.

[pageoneresults]

I don't pay much attention to WMT's indexed pages reporting. I've found WMT indexing reporting to be not reliable.

Hmmm, I'd pay real close attention to what GWT is reporting. I've yet to find it unreliable and I've used it for years. It has been responsible for early warning alerts for a variety of technical issues.

If Google is showing a large number of pages from your site and you don't have that many, it usually means one thing, you've got multiple paths to the same content and Google have most of them indexed. I see it happen on a regular basis.

Also, the one thing you want to pay special attention to are Crawl Errors, especially the Not Found ones. If your site has an excess of internal 404s, Google will whack you slowly. Then all of sudden, you'll wake up one day and see that most of your pages have been purged from the index. Once that happens to you, the next time GWT shows something out of kilter, you'll take action on it and not ignore it thinking that it is buggy. ;)

[aakk9999]

I have also noticed that when using site: command, of recently Google is reporting much more pages as being indexed. On one of domains we are following the number of indexed pages approximately doubled in the last few weeks.

Since the site is not very big (4000 pages aprox.) I was able to inspect more closely the results of site: command and noticed that the following URLs also appeared in the results:

- many pages that are blocked by robots.txt and have been blocked by robots.txt for a very long time now appear in the results. These pages were not there before and most of them are not linked from anywhere else than from within the site

- a number of pages that have had 301 redirect in place for over a year are now also listed in the site: results. Testing the redirect, the request still returns the correct HTTP 301 code. The page title reported in SERPs for these pages is link anchor followed by hyphen, followed by home page title

To me it almost looks like that Google has "folded in" some historic information on site URLs it knew about from way before.

In our case there is no impact on ranking or traffic but there is always a concern when you see number of "indexed pages" showing as double to what you know they should be.

With regards to GWT crawl errors, whilst giving a good guidance, I have noticed these not always being correct. For instance I have some page reported as "blocked by robots.txt" but when I copy this URL to "Crawler access" section of GWT it shows the page as "Allowed". Another instance I noticed is that Google is on occasions reporting some pages as 400 error. If you cut and paste URL shown in GWT, the page renders correctly. If you hover over URL google is showing in GWT, the correct URL is shown in status bar of the browser. But if you CLICK on the URL directly from GWT HTTP errors section, the in the URL that shows in the address bar and which is requested has all ? & and = replaced by %3F %3D and %26 and this throws server error. Now, why this is being reported for only handful of URLs and not across the board for all dynamic URLs is a mystery to me.

[indyank]

I am with aakk9999 on everything he has reported...thanks for neatly listing it down here...

I do suspect that the spiders sometime miss to read the robots.txt, resulting in it ignoring what is there...

I also often notice some key pages missing from the SERPS, as reported by others.When this happens, another page relating to the one that disappeared will be shown in the serps, but at much lower positions! May be page 3, 4 or 5 while the one that disappeared used to rank on page 1.

However, I will find that the overall traffic will remain more or less near what it should be.It might be one method how google is throttling traffic.

[netmeg]

I have one important page that has been popping in and out since August. No errors in WMT. PR4 as far as the toolbar is concerned. Tons of links to it. No reason as far as I can tell. It's a redirect (example.com/page.html redirects to example.com/word-word/) but that redirect has been in for at least three years and this only started happening in 2010. Right at the moment, it's back in and ranking high. I have no explanation, just marking it as yet another Google mystery. The content on the page is dynamically generated out of a database, and I have done NO tweaking on it whatsoever.

[Jonny6]

Google has been testing some new algos on a small subset of sites for some time, roughly since August I would guess? A very small number of sites have been hit hard with no explicable reason when compared to other sites in their niche. Some of these sites have returned in yo yo fashion, whilst others havnt.

I think we all may see the results of this quite soon, a big update is imminent?

[Jonny6]

For larger sites, the site operator query hasnt been accurate for a very long time, now its totally useless even when you include ommitted the results.

[jmccormac]

@netmeg And there I was thinking that you were a fan of Google. :) Seriously though, that comment about dynamically generated content is an interesting one. My main site uses a database and has about 350 million webpages. It seems to have been hit in mid December. It hasn't recovered yet. But the SERPs are interesting in that a pile of MFA scraper junk is now appearing ahead of links from legitimate and authoritative sites. Perhaps Google has banjaxed its algorithm trying to deal with mega linkswamps like ehow and somehow this has impacted all dynamic sites because of the way that Google had tried to implement its "solution".

Regards...jmcc