homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 38 message thread spans 2 pages: 38 ( [1] 2 > >     
Drastic drop in pages indexed - due to Big Daddy?

 6:24 pm on Feb 21, 2006 (gmt 0)

hey guys :)

site:domain gives me 130k pages, BD gives me 20k, but using the API always gave me around 10k, but yesterday i suddenly lost 8k pages!

anyone else experienced a huge drop in real indexed pages, as opposed to site:domain?

i read someone comment that his own opinion about sites losing pages was due to BD, from that comment i would assume others have also experienced the same as me.



 7:53 pm on Feb 21, 2006 (gmt 0)

I have a small, new site, and BD never had any more than 50% of my pages, non BD had about 75%.
As of today, BD has 100% and non-BD still has the same 75%

Give the Big Daddy time to catch up.. they've got more than a few pages to chew thru..



 8:53 pm on Feb 21, 2006 (gmt 0)

thanks for the reply.

yeah i'm not entirely bothered by BD but rather the 8k drop via the API.

i have another site with 150k urls in BD but only 2 in others, but teh API also shows only 2.


 11:00 pm on Feb 21, 2006 (gmt 0)

We have seen this as well - doing a "BigDaddy" IP search for "keyword site:www.example.com" gives 630 pages, but "keyword site:www.example.com/subdirectory/" gives 1200. So there are more pages found in just a subdirectory than on the site as a whole

The first search above gives 20000+ pages when searched on a non-"BigDaddy" IP address, and page-by-page spot checks shows less indexing/caching with "BigDaddy."


 7:21 pm on Feb 22, 2006 (gmt 0)

My site lost every single page in the site: search, but when searching for the main keyword, the site still showed up (on page 2 though, instead of early on page 1).

Yesterday there were 3 pages visible from site:, and today there are 2 pages and 10 links, and the position on page 1 is restored.

Total pages on the site is about 80.


 7:45 pm on Feb 22, 2006 (gmt 0)

On BD I show 355 page and about 100 of those are supplemental. On BD, the number of pages has gone from 450 to 890 to 450 to 350.

On non-BD I show 1,020 and the number continues to increase each week as pages are added to this website.

Googlbot is a constant visitor and I'm not the least bit worried about the page difference (other than the fact site search is horrible when it hits a BD DC).

I suspect that at a future date, there will be a massive update of the BD centers where a good amount of these missing pages will appear.


 3:33 am on Feb 23, 2006 (gmt 0)

I have 14,800 for 'site:' on BD, 10,000+ on others.

But when I search the individual urls on BD:

"Sorry, no information is available for the URL..."

Yet the page is still ranking in the serps (BD or otherwise)...?


 3:52 am on Feb 23, 2006 (gmt 0)

Are you using sitemaps?


 4:01 am on Feb 23, 2006 (gmt 0)

Yes, we have lost a ton of pages and the pages that have not been indexed are obviously not ranking. Without question, this is a problem with Big Daddy. So the question.. Is this temporary? Has anyone else heard any theories/explanations? This Big Daddy has really hurt our traffic.


 4:07 am on Feb 23, 2006 (gmt 0)

Are *you* using sitemaps?


 4:10 am on Feb 23, 2006 (gmt 0)

We have a Site Map but do not use "Google Sitemps"


 5:24 am on Feb 23, 2006 (gmt 0)

"Sorry, no information is available for the URL..."

Yet the page is still ranking in the serps (BD or otherwise)...?

How is this possible?

BTW This site has no G site map yet.


 7:13 am on Feb 23, 2006 (gmt 0)

The number of pages indexed for one of my domains have been increasing steadily since the beginning of Feb. till a few days ago when number of indexed pages suddenly dropped from 800+ to 240 on many of the DCs. Earlier tonight, the number of pages listed on,, and rose to around 700+ with 2/22/2006 cache dates while the BDs show older cache dates with either 240 or 1,300 pages indexed, latter containing around 1,200 supplementals.

I'm keeping my fingers crossed.


 7:24 am on Feb 23, 2006 (gmt 0)

This is strange.

You all report on fewer pages reported from BigDaddy, but try searching for site:.com on a BD vs. non-BD datacenters.

I see more pages in Bigdaddy. Not that many more, but more. Still, for my own domain I get 100 less pages on BD than on non-BD.


 9:14 am on Feb 23, 2006 (gmt 0)

google with BD seemed too wipe a big part of the index... which i think they didnt need to do... why couldnt they apply the new algo's to the existing pages they have indexed?

my page counts have dropped 1,000 % since BD's been propagating but finally are steadily increasing... my page counts went from 100k to just 800 pages then back up to 18,000 today.

I've noticed the same about my compeition and countless of other sites...


 9:16 am on Feb 23, 2006 (gmt 0)

it seems like BD is starting to catch up in index page counts vs the other dc's

it took BD a month after it seemed to have thrown out pages and re-index them again...


 4:26 pm on Feb 23, 2006 (gmt 0)

i'm not using sitemaps no, rely on crawling, old fashioned i guess :D

my site that lost many pages in BD and via API is 2 years old, currently 130k pages using site: in non bd dcs, 20k in BD and 2k using API

a new site, maybe 3 months old, is 2 pages only in non BD dcs, 170k pages in BD and 12k pages using API

Thew new site is increasing pages in BD, and until a few days ago showed 0 pages using API. This new site is clearly increasing, whilst the older site is dropping in BD and API.


 7:42 pm on Feb 27, 2006 (gmt 0)

Has anyone seen an improvement of their dropped pages? We have not, we seem to be losing more.
Anyone have insight, encouragement...?


 10:39 pm on Feb 27, 2006 (gmt 0)

No we do not but I have talked to several people who use them and do not and those that do have been dropped also.


 10:44 pm on Feb 27, 2006 (gmt 0)

Also losing pages

PR8 and PR7 pages...wired. The pages left are still ranking ok but missing alot of the main pages


 11:03 pm on Feb 27, 2006 (gmt 0)

yeah i've had no hit in rankings or visitors at all, just indexed pages

pages have crept up slightly, but every 5 minutes it keeps changing.


 2:15 pm on Feb 28, 2006 (gmt 0)

It seems liek I moved up for the pages that are still indexed, but there are also about 150,000 pages missing now which sucks.

Will this pages get reindexed?


 2:54 pm on Feb 28, 2006 (gmt 0)

>>>> Will this pages get reindexed? <<<<

This is the million dollar question....


 4:57 pm on Mar 1, 2006 (gmt 0)

Rookiecrd1 - my site is in the same position as yours - I suspect the increased ranking on the pages that do exist on BD is due to the pages that would normally rank above us also being missing.


 9:28 pm on Mar 1, 2006 (gmt 0)

incidently one of my site dropped from 98k to just 131 on site:www.mysite.com, the trafic doest seem to be affected but i guess its too early to tell.

i was just going throught these posts, can anyone give me a big daddy ip and a non big daddy ip so i can compare the difference.


 12:04 am on Mar 2, 2006 (gmt 0)

My site dropped dramatically today.

From 2.500.000 (Wednesday 22 February) now to 227.000 :-(


 12:41 am on Mar 2, 2006 (gmt 0)

After dropping from 250k to 74k in the last week, it just dropped again to 24k.

I hope this isn't permanant. I am really baffled. We are what I think an authority site, and now only have about 10% of out pages indexed.


 9:11 am on Mar 2, 2006 (gmt 0)

Now, I have only old supplementals in the index :-(


 10:04 am on Mar 2, 2006 (gmt 0)

It seems all the Google Engineers have left for the SES NYC in the midst of BigDaddy.
Many well to do old established sites are seeing the same sort problems, dropped from BD index and showing supplemental results in "site:" command.
Even I have been subjected to the same "searchquake". But to the utmost surprise there hasn't been a sharp decline in traffic, or hits from the robots, nor has there been any change in Pagerank.
It seems Matt and his team are totally unaware of this. The BigDaddy and its predecessors are seriously causing panic for many.
Can someone please tell, how long should I hold my patience? Of course there is no other way out also other than waiting for things to settle permanently.


 10:06 am on Mar 2, 2006 (gmt 0)

Yes, all Google important engineers probably are at SES - aslong as Google blackhat friends are kept happy they dont really worry about the whitehats.

A happy blackhat does less damage therefore Google seems to do everything to please them at the conferences etc.

As to the rest of us webmasters - they dont really care.

This 38 message thread spans 2 pages: 38 ( [1] 2 > >
Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved