Drastic drop in pages indexed - due to Big Daddy?

[leoo24]

hey guys :)

site:domain gives me 130k pages, BD gives me 20k, but using the API always gave me around 10k, but yesterday i suddenly lost 8k pages!

anyone else experienced a huge drop in real indexed pages, as opposed to site:domain?

i read someone comment that his own opinion about sites losing pages was due to BD, from that comment i would assume others have also experienced the same as me.

[Tearabite]

I have a small, new site, and BD never had any more than 50% of my pages, non BD had about 75%.
As of today, BD has 100% and non-BD still has the same 75%

Give the Big Daddy time to catch up.. they've got more than a few pages to chew thru..

.

[leoo24]

thanks for the reply.

yeah i'm not entirely bothered by BD but rather the 8k drop via the API.

i have another site with 150k urls in BD but only 2 in others, but teh API also shows only 2.

[insight]

We have seen this as well - doing a "BigDaddy" IP search for "keyword site:www.example.com" gives 630 pages, but "keyword site:www.example.com/subdirectory/" gives 1200. So there are more pages found in just a subdirectory than on the site as a whole

The first search above gives 20000+ pages when searched on a non-"BigDaddy" IP address, and page-by-page spot checks shows less indexing/caching with "BigDaddy."

[ctor]

My site lost every single page in the site: search, but when searching for the main keyword, the site still showed up (on page 2 though, instead of early on page 1).

Yesterday there were 3 pages visible from site:, and today there are 2 pages and 10 links, and the position on page 1 is restored.

Total pages on the site is about 80.

[BillyS]

On BD I show 355 page and about 100 of those are supplemental. On BD, the number of pages has gone from 450 to 890 to 450 to 350.

On non-BD I show 1,020 and the number continues to increase each week as pages are added to this website.

Googlbot is a constant visitor and I'm not the least bit worried about the page difference (other than the fact site search is horrible when it hits a BD DC).

I suspect that at a future date, there will be a massive update of the BD centers where a good amount of these missing pages will appear.

[nzmatt]

I have 14,800 for 'site:' on BD, 10,000+ on others.

But when I search the individual urls on BD:

"Sorry, no information is available for the URL..."

Yet the page is still ranking in the serps (BD or otherwise)...?

[Steph_R]

leoo24,
Are you using sitemaps?

[dinnerware]

Yes, we have lost a ton of pages and the pages that have not been indexed are obviously not ranking. Without question, this is a problem with Big Daddy. So the question.. Is this temporary? Has anyone else heard any theories/explanations? This Big Daddy has really hurt our traffic.

[Steph_R]

dinnerware,
Are *you* using sitemaps?

[dinnerware]

We have a Site Map but do not use "Google Sitemps"

[nzmatt]

"Sorry, no information is available for the URL..."

Yet the page is still ranking in the serps (BD or otherwise)...?

How is this possible?

BTW This site has no G site map yet.

[Halfdeck]

The number of pages indexed for one of my domains have been increasing steadily since the beginning of Feb. till a few days ago when number of indexed pages suddenly dropped from 800+ to 240 on many of the DCs. Earlier tonight, the number of pages listed on 216.239.59.147, 64.233.183.104, and 66.102.9.104 rose to around 700+ with 2/22/2006 cache dates while the BDs show older cache dates with either 240 or 1,300 pages indexed, latter containing around 1,200 supplementals.

I'm keeping my fingers crossed.

[Nikke]

This is strange.

You all report on fewer pages reported from BigDaddy, but try searching for site:.com on a BD vs. non-BD datacenters.

I see more pages in Bigdaddy. Not that many more, but more. Still, for my own domain I get 100 less pages on BD than on non-BD.

[dakman]

google with BD seemed too wipe a big part of the index... which i think they didnt need to do... why couldnt they apply the new algo's to the existing pages they have indexed?

my page counts have dropped 1,000 % since BD's been propagating but finally are steadily increasing... my page counts went from 100k to just 800 pages then back up to 18,000 today.

I've noticed the same about my compeition and countless of other sites...

[dakman]

it seems like BD is starting to catch up in index page counts vs the other dc's

it took BD a month after it seemed to have thrown out pages and re-index them again...

[leoo24]

i'm not using sitemaps no, rely on crawling, old fashioned i guess :D

my site that lost many pages in BD and via API is 2 years old, currently 130k pages using site: in non bd dcs, 20k in BD and 2k using API

a new site, maybe 3 months old, is 2 pages only in non BD dcs, 170k pages in BD and 12k pages using API

Thew new site is increasing pages in BD, and until a few days ago showed 0 pages using API. This new site is clearly increasing, whilst the older site is dropping in BD and API.

[dinnerware]

Has anyone seen an improvement of their dropped pages? We have not, we seem to be losing more.
Anyone have insight, encouragement...?

[dinnerware]

No we do not but I have talked to several people who use them and do not and those that do have been dropped also.

[powerstar]

Also losing pages

PR8 and PR7 pages...wired. The pages left are still ranking ok but missing alot of the main pages

[leoo24]

yeah i've had no hit in rankings or visitors at all, just indexed pages

pages have crept up slightly, but every 5 minutes it keeps changing.

[rookiecrd1]

It seems liek I moved up for the pages that are still indexed, but there are also about 150,000 pages missing now which sucks.

Will this pages get reindexed?

[dinnerware]

>>>> Will this pages get reindexed? <<<<

This is the million dollar question....

[Wibfision]

Rookiecrd1 - my site is in the same position as yours - I suspect the increased ranking on the pages that do exist on BD is due to the pages that would normally rank above us also being missing.

[dopecoder]

incidently one of my site dropped from 98k to just 131 on site:www.mysite.com, the trafic doest seem to be affected but i guess its too early to tell.

i was just going throught these posts, can anyone give me a big daddy ip and a non big daddy ip so i can compare the difference.

[Armi]

My site dropped dramatically today.

From 2.500.000 (Wednesday 22 February) now to 227.000 :-(

[rookiecrd1]

After dropping from 250k to 74k in the last week, it just dropped again to 24k.

I hope this isn't permanant. I am really baffled. We are what I think an authority site, and now only have about 10% of out pages indexed.

[Armi]

Now, I have only old supplementals in the index :-(

[seochristine]

It seems all the Google Engineers have left for the SES NYC in the midst of BigDaddy.
Many well to do old established sites are seeing the same sort problems, dropped from BD index and showing supplemental results in "site:" command.
Even I have been subjected to the same "searchquake". But to the utmost surprise there hasn't been a sharp decline in traffic, or hits from the robots, nor has there been any change in Pagerank.
It seems Matt and his team are totally unaware of this. The BigDaddy and its predecessors are seriously causing panic for many.
Can someone please tell, how long should I hold my patience? Of course there is no other way out also other than waiting for things to settle permanently.

[Dayo_UK]

Yes, all Google important engineers probably are at SES - aslong as Google blackhat friends are kept happy they dont really worry about the whitehats.

A happy blackhat does less damage therefore Google seems to do everything to please them at the conferences etc.

As to the rest of us webmasters - they dont really care.