Pages Dropping Out of Big Daddy Index

[GoogleGuy]

Continued from: [webmasterworld.com...]

---

One thing to bear in mind is that Bigdaddy will have different crawl priorities. That can account for some of it. If you've run into any spam problems in the past, you might also want to do a reinclusion request. Otherwise, please send an email to bostonpubcon2006 at gmail.com with the subject line "crawlpages" (all one word), and I'll ask someone to see if they notice any commonalities.

[Lorel]

Re Google possibly using historical data to prove who owns websites--I think this is a good possiblity as the site I reported earlier that had lost pages was previously owned by someone else.

Out of 32 sites I manage this is the only one that is having this problem. All my other clients bought their domain brand new.

Also, the index for this site keeps going up and down. Not settled yet.

[youfoundjake]

youfoundjake
How about this one?

[64.233.161.104...]

Nothing there either.. bah google is a fad... Just kidding. Im happy that the one page is there, enough information is on it to let the visitor know what the site is for.

[texasville]

Just to clarify, I think what is happening will only affect sites that have had redesigns, dup content problems, scraped pages, changed urls, failed 301's, 302's, hijacking and so forth. If this theory is correct all pages should return to sites as long as they belong there. If your pages were scraped by some other site, their pages should NEVER return. In other words, google is hoping to end hijacking and piracy forever. However, it seems that changes you made in your own site may trigger this same filter, or algo, or infrastructure or whatever you want to call it.

[Stefan]

Just to clarify, I think what is happening will only affect sites that have had redesigns, dup content problems, scraped pages, changed urls, failed 301's, 302's, hijacking and so forth.

I think you're bang-on with that. Essentially, those sites that didn't have problems before BD, didn't have problems after it (disappearing pages). The problems beforehand might not have been recognized as such until BD spread across the DC's, but they were there all the same. If you have a site that wasn't affected, you can see it when you read the posts, and you covered it quite well in the quote above (although "redesigns" should be changed to "major redesigns that triggered the Sandbox").

[g1smd]

The "experimental" results that were on 72.14.207.99 for the last few weeks, spread to a few other datacentres yesterday and to a lot more today.

The results are VERY different. This is the biggest change I have seen in Google results in at least 5 years of looking. I want to start calling this an update.

The results are utterly awful. There is no "exact match" for "quoted searches" any more, either.

Some of the searches I do, now have thousands of results, rather than dozens, but none of the results actually fit the search query.

Many SERPs are stuffed full of supplementals. Some are 100% supplemental results.

Some sites have lost 99% of their indexed pages.

Many cache dates go back to 2004 January.

[texasville]

Could be an update. Could be much more than an update. But that is all perception anyway. So...what is the name? We had jagger, bourbon, florida, others...dang!

[MLHmptn]

BIG DADDY! since it was never completed and because of the massive changes! But who knows, maybe it's still not an update. Maybe we're just seeing a step in the new infrastructure.

[edited by: MLHmptn at 3:36 am (utc) on May 3, 2006]

[youfoundjake]

On a side note, the DC located at [66.249.93.104...]
has 2 pages of mine, both index pages. one is www.sitename.com, and the other is sitename.com
And the kicker is that both the pages are completely different becaused I changed the whole site layout.
The list with www is the most recent version, can someone help me understand this?

[texasville]

>>>>And the kicker is that both the pages are completely different becaused I changed the whole site layout. <<<<

Read thru the last couple of pages of this thread and you might get an idea of just a "suggested" theory. Are these the only two pages you have in that dc?

You might check [72.14.203.99...] and see what it shows you...just out of curiosity.

[jtoddv]

I think what is happening will only affect sites that have had redesigns, dup content problems, scraped pages, changed urls, failed 301's, 302's, hijacking and so forth.

I couldn't agree more... this is exactly what I am seeing as well. Plus what also makes me seem to think this is correct is the comment from GG:

If you've run into any spam problems in the past, you might also want to do a reinclusion request.

I redid my url rewrites about 6 months back and 301'ed all old pages to their new corresponding pages. I had 3,800 pages in the index now down to between 100-400 as it fluxuates. I am seeing all of the previous urls listed as supplementals and most of the new ones are missing. Plus I had dupe content on my site. This was recitifed over a year ago, but the other site in question is in the same boat as my main site.

I have yet to submit my site for a reinclusion request as GG suggested. Nor have I submitted the 'crawlpages' request either. Has anybody that submitted either of these seen any headway?

Also we recently placed a 301 on a very highly ranked site (PR 7)... and it has still yet to change in the index, when if my memory serves me correctly would have taken no more than two weeks to change over... maybe I am wrong on that?

[youfoundjake]

Read thru the last couple of pages of this thread and you might get an idea of just a "suggested" theory. Are these the only two pages you have in that dc?

You might check [72.14.203.99...] and see what it shows you...just out of curiosity.

same result, only 2 pages found
i also noticed that the mysite.com is the older version, while www.mysite.com is the newer version. i'm not sure what you guys mean by supplementals, but mysite.com only shows up and i have to click the link to repeat the search with the omitted results included which then shows me my newer version. sigh..

[mgrohan]

OK my i am having the same problem as a few other people in indexed pages in google dropping to just the single index page. The things is this can be seen on ALL google datacenters i have checked.

My main page is PR4 my inner pages are PR2-3, I have backlinks on inner pages, but these pages are not showing on google.
MSN currently shows: 362 pages and Yahoo: 190 pages.

What can the reasons for google dropping me down to just my homepage? The only changes i have made in the time before this change is making some frontpage outbound links - rel='no follow'.

There may be some dup content with mod rewritten forum and directory, but this doesn;t effect other pages does it?

What the best thing to do in this situation. Should i start changing things on my website? if so what?

Thanks - desperate for help...

[youfoundjake]

And just to through a little humor into it, these past 2 sites have 2 times as many pages listed as the other data centers.

[bkpix]

Here's the situation: I created a small niche site last year, uploaded a few pages in spring 2005, and uploaded about 50 pages of content in March 2006. At that point I submitted the site to Google. I have a Google sitemap that is up to date. All content is original, written by me.

In early April doing a site:www.mysite.com search showed about 40 pages: not quite the full site, but close. Then about mid April the same search produced 13 pages, even though I now have about 85 on the site. As of last week, Google shows only two pages. They are:

www.mysite.com
www.mysite.com/index.php

which are, of course, the same page.

Keeping in mind I am new to this stuff -- I have read a lot of posts about disappearing pages and Big Daddy but remain pretty confused -- my question is, what can I do to get pages back into Google?

MSN and Yahoo seem to find them just fine.

Thanks for any advice.

[tigger]

I lost another 3 pages over night and still not getting anything new in

>my question is, what can I do to get pages back into Google?

suppose like most here hope its a G bug and wait for a fix

[walkman]

Many of my pages coming back, and Google is indexing relatively fast. Will hold on celebrating though, as this has happened before, only to be disappointed.

[Relevancy]

Some of mine are coming back too, but they are all the dropped supplemental ones though.. doesnt help

[Relevancy]

Could similar description tags trigger this dup filter thing-a-ma-bobber/ algo change/ whatever? If so that is lame.

[LuckyGuy]

dazzlindonna,

did you change anyything on our side last year? So we can get closer to that suggestion of beeing trapped into a redesign-filter?

I donīt know if itīs this but i redesigned my pages in midī05 i did for all of them but not all are vanished. Vanished are only the sides that have link-level 3, so itīs detail-pages with very not similar content but same layout. If you trigger a filter by having the same layout on many pages but different content that would be very bad.
Today I got 10 pages back out of 5800. Stay on round about 1300 like a rock. Half of them does not exist, some are old Cache pages, the little rest are OK.

[wheelie34]

Total pages for site = approx 970

Monday 1st May site: = 367
Tuesday 2nd " site: = 410
Wednesday 3rd site: = 407

All average figures over all dc's so theres some movement happening

[tigger]

I don't see how a similar description tags could trigger this, after all if you have a site/store on XYZ products you could very feasibly have pages with descriptions like ZXY products 1/2/3/4 and so on

[LuckyGuy]

tigger,

that is what i meant.Waht if this Dup Content or similar-Filter works not on content but on page layout? Creepily!

[tigger]

I really can't see how this dup filter would be on page layout after all that would wipe out the bulk of sites!

hold on inst that what G has done!

[LuckyGuy]

tigger,

it has wiped out the bulk of the sides!
All gone. I donīt say it just page layout but it could have something to do with it.

[RichTC]

They are still not cashing new content pages.

Following the BD roll out imo they applied some sort of extra filter to stop new content pages being cashed quickly even on high PR sites.

On one PR6 site im still waiting for google to cashe new additional pages listed on a category page on the site for over three weeks now. The Category page is cashed on a regular basis, just google isnt following any of the links off it to the new content.

In another example, of about 60 new pages listed in a similar way, the google bot has followed some of the links to the new pages, but not all of them.

Im not sure if the new bot doesnt work properly or if its just a case that its set to sandbox more new pages prior to listing

[tigger]

same here, new pages linked from the index and with a index cached date of 27/4 still not pulling the new stuff in

[vanessa19]

I have been checking and the pages (approx 100) dropped for my site are ones which have a duplicate description which I used as a templete page for the order forms and although the content is different I didn't change the key words or the description tag.

[cbartow]

All the supplements dropped for a few of my sites again this morning. Moved from 800->600->80 and now a whole 18 pages.

Maybe June will be a better month.

[g1smd]

There are several types of duplicate content that can affect a site:

If you have both www and non-www pages listed then get the 301 redirect installed immediately. If you have several domains, perhaps .com and .org and .co.uk, etc, then make sure you 301 redirect to the one that you want to be listed.

If the title and meta description are not unique per page, Google will filter the extra pages out. Make sure that every page has a unique title and meta description and that they describe exactly what is on the page that they are placed on.

Also make sure that each page has only one URL that can reach it. Use Xenu LinkSleuth to check your site out for that too.

[edited by: g1smd at 12:49 pm (utc) on May 3, 2006]

[prieshach]

Previously when I did a site: search for my site, Google would show the meta description as the snippet if there was one.

Now, for pages cached April 28 and later, it no longer shows the meta even if there is one (72.14.203.104). Pages cached pre-28 still show the meta but 28 and after do not.