Welcome to WebmasterWorld Guest from 50.19.0.90

Message Too Old, No Replies

Drastic drop in pages indexed - due to Big Daddy?

     
6:24 pm on Feb 21, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 6, 2003
posts:164
votes: 0


hey guys :)

site:domain gives me 130k pages, BD gives me 20k, but using the API always gave me around 10k, but yesterday i suddenly lost 8k pages!

anyone else experienced a huge drop in real indexed pages, as opposed to site:domain?

i read someone comment that his own opinion about sites losing pages was due to BD, from that comment i would assume others have also experienced the same as me.

7:53 pm on Feb 21, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 21, 2006
posts:436
votes: 0


I have a small, new site, and BD never had any more than 50% of my pages, non BD had about 75%.
As of today, BD has 100% and non-BD still has the same 75%

Give the Big Daddy time to catch up.. they've got more than a few pages to chew thru..

.

8:53 pm on Feb 21, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 6, 2003
posts:164
votes: 0


thanks for the reply.

yeah i'm not entirely bothered by BD but rather the 8k drop via the API.

i have another site with 150k urls in BD but only 2 in others, but teh API also shows only 2.

11:00 pm on Feb 21, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:May 4, 2004
posts:44
votes: 0


We have seen this as well - doing a "BigDaddy" IP search for "keyword site:www.example.com" gives 630 pages, but "keyword site:www.example.com/subdirectory/" gives 1200. So there are more pages found in just a subdirectory than on the site as a whole

The first search above gives 20000+ pages when searched on a non-"BigDaddy" IP address, and page-by-page spot checks shows less indexing/caching with "BigDaddy."

ctor

7:21 pm on Feb 22, 2006 (gmt 0)

Inactive Member
Account Expired

 
 


My site lost every single page in the site: search, but when searching for the main keyword, the site still showed up (on page 2 though, instead of early on page 1).

Yesterday there were 3 pages visible from site:, and today there are 2 pages and 10 links, and the position on page 1 is restored.

Total pages on the site is about 80.

7:45 pm on Feb 22, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member billys is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 1, 2004
posts:3181
votes: 0


On BD I show 355 page and about 100 of those are supplemental. On BD, the number of pages has gone from 450 to 890 to 450 to 350.

On non-BD I show 1,020 and the number continues to increase each week as pages are added to this website.

Googlbot is a constant visitor and I'm not the least bit worried about the page difference (other than the fact site search is horrible when it hits a BD DC).

I suspect that at a future date, there will be a massive update of the BD centers where a good amount of these missing pages will appear.

3:33 am on Feb 23, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Mar 17, 2004
posts:327
votes: 0


I have 14,800 for 'site:' on BD, 10,000+ on others.

But when I search the individual urls on BD:

"Sorry, no information is available for the URL..."

Yet the page is still ranking in the serps (BD or otherwise)...?

3:52 am on Feb 23, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 10, 2006
posts:68
votes: 0


leoo24,
Are you using sitemaps?
4:01 am on Feb 23, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 27, 2003
posts:55
votes: 0


Yes, we have lost a ton of pages and the pages that have not been indexed are obviously not ranking. Without question, this is a problem with Big Daddy. So the question.. Is this temporary? Has anyone else heard any theories/explanations? This Big Daddy has really hurt our traffic.
4:07 am on Feb 23, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 10, 2006
posts:68
votes: 0


dinnerware,
Are *you* using sitemaps?
4:10 am on Feb 23, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 27, 2003
posts:55
votes: 0


We have a Site Map but do not use "Google Sitemps"
5:24 am on Feb 23, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Mar 17, 2004
posts:327
votes: 0


"Sorry, no information is available for the URL..."

Yet the page is still ranking in the serps (BD or otherwise)...?

How is this possible?

BTW This site has no G site map yet.

7:13 am on Feb 23, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Nov 10, 2005
posts:240
votes: 0


The number of pages indexed for one of my domains have been increasing steadily since the beginning of Feb. till a few days ago when number of indexed pages suddenly dropped from 800+ to 240 on many of the DCs. Earlier tonight, the number of pages listed on 216.239.59.147, 64.233.183.104, and 66.102.9.104 rose to around 700+ with 2/22/2006 cache dates while the BDs show older cache dates with either 240 or 1,300 pages indexed, latter containing around 1,200 supplementals.

I'm keeping my fingers crossed.

7:24 am on Feb 23, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:May 13, 2003
posts:442
votes: 0


This is strange.

You all report on fewer pages reported from BigDaddy, but try searching for site:.com on a BD vs. non-BD datacenters.

I see more pages in Bigdaddy. Not that many more, but more. Still, for my own domain I get 100 less pages on BD than on non-BD.

9:14 am on Feb 23, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 2, 2004
posts:91
votes: 0


google with BD seemed too wipe a big part of the index... which i think they didnt need to do... why couldnt they apply the new algo's to the existing pages they have indexed?

my page counts have dropped 1,000 % since BD's been propagating but finally are steadily increasing... my page counts went from 100k to just 800 pages then back up to 18,000 today.

I've noticed the same about my compeition and countless of other sites...

9:16 am on Feb 23, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 2, 2004
posts:91
votes: 0


it seems like BD is starting to catch up in index page counts vs the other dc's

it took BD a month after it seemed to have thrown out pages and re-index them again...

4:26 pm on Feb 23, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 6, 2003
posts:164
votes: 0


i'm not using sitemaps no, rely on crawling, old fashioned i guess :D

my site that lost many pages in BD and via API is 2 years old, currently 130k pages using site: in non bd dcs, 20k in BD and 2k using API

a new site, maybe 3 months old, is 2 pages only in non BD dcs, 170k pages in BD and 12k pages using API

Thew new site is increasing pages in BD, and until a few days ago showed 0 pages using API. This new site is clearly increasing, whilst the older site is dropping in BD and API.

7:42 pm on Feb 27, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 27, 2003
posts:55
votes: 0


Has anyone seen an improvement of their dropped pages? We have not, we seem to be losing more.
Anyone have insight, encouragement...?
10:39 pm on Feb 27, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 27, 2003
posts:55
votes: 0


No we do not but I have talked to several people who use them and do not and those that do have been dropped also.
10:44 pm on Feb 27, 2006 (gmt 0)

Full Member

10+ Year Member

joined:May 24, 2002
posts:284
votes: 0


Also losing pages

PR8 and PR7 pages...wired. The pages left are still ranking ok but missing alot of the main pages

11:03 pm on Feb 27, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 6, 2003
posts:164
votes: 0


yeah i've had no hit in rankings or visitors at all, just indexed pages

pages have crept up slightly, but every 5 minutes it keeps changing.

2:15 pm on Feb 28, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 8, 2004
posts:142
votes: 0


It seems liek I moved up for the pages that are still indexed, but there are also about 150,000 pages missing now which sucks.

Will this pages get reindexed?

2:54 pm on Feb 28, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 27, 2003
posts:55
votes: 0


>>>> Will this pages get reindexed? <<<<

This is the million dollar question....

4:57 pm on Mar 1, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 15, 2005
posts:103
votes: 0


Rookiecrd1 - my site is in the same position as yours - I suspect the increased ranking on the pages that do exist on BD is due to the pages that would normally rank above us also being missing.
9:28 pm on Mar 1, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:May 12, 2005
posts:45
votes: 0


incidently one of my site dropped from 98k to just 131 on site:www.mysite.com, the trafic doest seem to be affected but i guess its too early to tell.

i was just going throught these posts, can anyone give me a big daddy ip and a non big daddy ip so i can compare the difference.

12:04 am on Mar 2, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:May 28, 2004
posts:89
votes: 0


My site dropped dramatically today.

From 2.500.000 (Wednesday 22 February) now to 227.000 :-(

12:41 am on Mar 2, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 8, 2004
posts:142
votes: 0


After dropping from 250k to 74k in the last week, it just dropped again to 24k.

I hope this isn't permanant. I am really baffled. We are what I think an authority site, and now only have about 10% of out pages indexed.

9:11 am on Mar 2, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:May 28, 2004
posts:89
votes: 0


Now, I have only old supplementals in the index :-(
10:04 am on Mar 2, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 17, 2005
posts:41
votes: 0


It seems all the Google Engineers have left for the SES NYC in the midst of BigDaddy.
Many well to do old established sites are seeing the same sort problems, dropped from BD index and showing supplemental results in "site:" command.
Even I have been subjected to the same "searchquake". But to the utmost surprise there hasn't been a sharp decline in traffic, or hits from the robots, nor has there been any change in Pagerank.
It seems Matt and his team are totally unaware of this. The BigDaddy and its predecessors are seriously causing panic for many.
Can someone please tell, how long should I hold my patience? Of course there is no other way out also other than waiting for things to settle permanently.

Dayo_UK

10:06 am on Mar 2, 2006 (gmt 0)

Inactive Member
Account Expired

 
 


Yes, all Google important engineers probably are at SES - aslong as Google blackhat friends are kept happy they dont really worry about the whitehats.

A happy blackhat does less damage therefore Google seems to do everything to please them at the conferences etc.

As to the rest of us webmasters - they dont really care.

This 38 message thread spans 2 pages: 38
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members