Some big observations on dropped pages

[tsm26]

I have been trying to figure our why my site dropped from 57,000 pages down to only 700. Today I noticed a huge pattern, and barring something major, I believe it is the reason for the dropped pages. First, I noticed that all pages three levels deep and higher are indexed. Any pages indexed lower than that are externally linked in some way.

How I noticed this, is that we have a huge directory of content arranged alphabetically with each letter being a seperate page a.html for example. From my front page I have a.html linked, and then all the content links on that page. The content that starts with a letter 'a' is all indexed. The pages like b.html and c.html are also indexed, but the individual content pages aren't.

So, what this means is that Google is giving an overall site PR which tells it how many levels down it will index. In my limited research it seems that a site with a front page of PR 5 will get indexed three levels down, and a site of PR 6 will get indexed four levels down. Those below PR 5 I have looked at are barely getting spidered.

When doing this, keep in mind that your front page counts as a level. So if you are only PR 5 it seems like if you have a huge directory don't split it up into sections, just have a huge page with the links to it all. This of course totally hoses usability but you will get spidered.

Also, externally linked pages will get spidered, as a few of the pages listed under the other letters are indexed, as they are linked in blogs and other sites. This is across the board what is happening on my site and the others I have looked at.

Count your levels getting spidered and you will notice how deep they are going. For me, three levels and that is it except for the externally linked individual pages I have seen.

[edited by: tedster at 6:16 pm (utc) on May 22, 2006]
[edit reason] formatting [/edit]

[webtress]

pageoneresults - you are on the money as always :) I truly enjoy reading your postings.

[trinorthlighting]

I am doing an experiment with my dropped pages. I just joined adsense and put advertisements on my dropped pages to see if they show back up in the cache again. I will keep you posted.

[sandpetra]

My pages have bounced back. Just as i had rewritten the navigation last night sigh. The only thing I did recently was to get as many links to inner pages as I could from external sites.

Was this the reason for the pages back in the index? I knew that some of my pages still had pr, even though they were not visible using site: command.

[cbartow]

trinorthlighting:

I tried putting adsense on and it didn't do a thing. The mediabot visited a bunch of pages and none got back into the index.

I have another site that has adsense running on it the entire time and is still suffering.

[trinorthlighting]

How long have you done this? I just put my adsense on today. Now at least googlebot is visiting it

[Atomic]

I just put my adsense on today. Now at least googlebot is visiting it

Every time I make a change and there is a result I seriously consider that my actions had nothing to do with the result. Don't be so sure that what you did made Googlebot swing by.

[trinorthlighting]

Well, googlebot was not visiting my pages before. If you read matts blog he stated that the media bot will assist googlebot. I noticed both have been visiting pages they never have before. I am going to see what happens next week.

[Atomic]

If you read matts blog he stated that the media bot will assist googlebot.

If this is true it then I have that much less respect for those people. So if you are willing to put AdSense on your pages they are more willing to send Googlebot to your site? In other words, if you use Google's advertising and not a competitor they are willing to tease you with the possibility of being indexed by swinging Googlebot to your site. If you don't use Google's AdSense system then they are less likely to index you. That seems to be what you are saying Google is implying. Once you think about it.

[tedster]

I think the point is that if you have Adsense, then mediabot's spidering will no longer be duplicated by regular googlebot becuase now thyey can share a cache. No advantage at all in your rankings or indexing, just a savings in bandwidth for both you and Google.

[bigdrummer]

After having full recovery (all pages in main index) on a hyphenated domain things have gone goofy again in past 2 days. Indexed pages have gone from 414 to 80'ish (across the DCs). Also seeing old pages which no longer exist showing up as supplementals. Anyone noticed a 'swing to the left' in terms of going back into bad territory? I made no changes in between, but perhaps this type of blip happens during the monthly crawl?

[Whitey]

For the results of a verified experiment on "dropped pages" involving the treatment of Google's indexing of pages containing unrelated on page links look here:

[webmasterworld.com...]

I'd say a large proportion of webmasters are in this trap.

[tsm26]

I don't think that article is very relevant to this particular discussion and it also brings up big problems if it is true. Take an encyclopedia site for example. What is it's theme? Wikipedia is probably the best crawled site on the Internet, and it doesn't have a central theme. Our dropped content was straight from a business encyclopedia that we licensed and is the only online publishing of it. Not one page of ours shows up as a supplemental result that I can find, and this has never been our problem. The pages just simply were dropped excluding 10 different ones that were directly linked from Wikipedia and blogs. I think as these number of links grow the others will follow. Really though, the analysis of that has a ton of holes and problems associated with it, and I doubt that Google would put itself into that situation.

[Whitey]

tsm26 - Google has been working for many years trying to get the area of themes improved.

The problem is clearly identified on Matt Cutt's blog. When you have links on gambling, viagra , underwear , fast cash appearing on a travel site which contains travel content, Google is able to identify the offending page.

The page disappears, the links to your site disappear and the whole thing propogates through the network.

Our experiment proves it with pinpoint accuracy and the very large no. of sites and the [ network in question ] recognise it.

However, it is only one of Google's many assault's on SPAM , so those need to be taken into account also , plus ,.... i firmly believe there is collataral damage.

[edited by: Whitey at 2:49 pm (utc) on June 10, 2006]

[Whitey]

tsm26 - Google has been working for many years trying to get the area of themes improved.

The problem is clearly identified on Matt Cutt's blog. When you have links on gambling, viagra , underwear , fast cash appearing on a travel site which contains travel content, Google is able to identify the offending page.

The page disappears, the links to your site disappear and the whole thing propogates through the network.

If your page offends the linking guidelines, your pages disappears as well as your overall ranking previously supported by those pages.

Our experiment proves it with pinpoint accuracy and the very large no. of sites and the [ network we observed in question ] recognise it - and there are some good big sites in the network - and spammy ones that should be removed -it's those one's which are causing the problem.

However, it is only one of Google's many assault's on SPAM causing page drops.... and i firmly believe there is collataral damage as Google irons out it's problems, not only related to this alert.

[tsm26]

We have no incoming links from anything like the problems others see. Only have two affiliates and are a venture business website respected and used by many vc groups and others. We have a ton of original content and are well indexed other than our deep pages. It isn't that big of a deal. We are well seated in Google with 6k pages indexed. I do think this whole thing is a big circle of removing spammy pages that reach a lot of non offenders. That said, I already know of four website owners that are now fully indexed by putting up sitemaps like is the main context of this thread, getting lower levels indexed. The causes behind it are secondary when you don't have many problems on your own site. We need to discuss things we can change.

[Whitey]

tsm26 - So what happens to you when a site that links to you [ and supports your PR & keywords ] looses it's page, because it contains a SPAM link?

You loose your position, even if it's 3 or 4 sites removed - it travels down the line.

Then you look at the principles behind the algo. and possibly you have pages that fall into the same definition of the algo. applied to legitimate sites, such as your, and then loose pages.

I don't know the 2nd part of it, but some experiments on this would likely reveal combinations of linking and content issues - even of an internal nature [ like "link islands" ].

However, I keep emphasising that Google does appear to have a lot of other technical errors occurring, which is confusing people, especially who are making observations via the disfuntional site: tool [ possibly still not fully fixed ]

Sorry, this is on this thread, but i think there's an overlap, which probably should now go to the other thread - but there is a synergy and need, i felt to alert folks of the *urgent* propogation component.

Amazingly, few people have participated in the discussion [ and also the discussion of the related network's forum ] - despite it being recognised as a major collapse issue by the administrators of this key network. Here it is again [webmasterworld.com...]

[dibbern2]

<thread fatigue=20pages>

time to move on

[rekitty]

Before this thread dies out, I want to send a BIG thank you to tsm26 and to ask you one question.

My site dropped from nearly a million pages to less than 100,000 indexed by google overnight. I saw a similar drop in a competitor too. tsm26 did an amazing job of quickly identifying the issue and trying a solution that ended up solving his problem. Too few folks are willing to try innovative solutions like you did because they are counter to SEO folklore. Nice work!

tsm26, one final question for you: after implementing your solution did you get hurt in the results for the higher up pages that probably target more competitive terms, ie your pages that remained in the index when your page count dropped?

Our remaining pages currently rank very well for some very competitive terms. I'm concerned about changing the site structure to get the more obscure pages indexed will suck PR from our competitive pages and they will drop in the serps.

I'll try to make my question easy for you with multiple choice answers:

A. We never ranked for any competitive terms, so the change didn't matter.
B. We ranked well for competitive terms and still continue to after the change.
C. We ranked well for competitive terms and those terms got hurt in the serps after the change but the traffic from the more obscure terms more than made up for that loss.
D. Something else. Please explain :-)

Thanks again!

[tsm26]

We did not rank that well for many competitive keywords on Google as we are only a year old. The traffic from the lower pages more than made up for any loss from the higher pages getting diluted. If you have good content, the lower content should be where you want them to go anyway. Of course it all depends on the site.

[mike1256]

so let me see if i have this right

if i sell a green widget on my red widget websites domain, my green widget pages will be dropped because the websites about red widgets?

< continued here: [webmasterworld.com...] >

[edited by: tedster at 5:02 pm (utc) on June 14, 2006]