Number of pages indexed drops - a lot and very fast.

Why...

[Cyclob]

My page index has decrease more than 100% since BigDaddy hit. And this morning I found that it continue to drop from 28,600 to now 7,200 pages.

Anyone know the reasons why? I just change the redirection method within my site from 302 to 301. Also already create Google sitemap and status of sitemap is OK.

I'm pretty sure that I didn't use any bad technique to fool GG, everything that I do was all white technique.

Anyone has the same problem? I'm now is very concern about this issue.

Thanks for your suggesstion in advance.

[tedster]

I'm hearing a lot of reports like this right now, as Google's Big Daddy infrastructure is in the middle of rolling out -- but not yet stable. Here's the important question -- is your traffic affected? If it isn't, I'd just wait it out.

But if your traffic is affected, and pages that used to get search traffic now are not, or are severely depressed, then it's time for a through check-up of your site. These threads have lots of detail about areas to inspect for problems:

Checklist for Sudden Drops in Rank [webmasterworld.com]
Dropped from Google - a checklist to find out why [webmasterworld.com]
Dropped Site Checklist [webmasterworld.com]

[sit2510]

I'm experiencing the same things too. My pages began to drop from BD database since the beginning of last week, but no problem with other non-BD databases. Because BD is spreading to more and more datacenters, my traffic began to decline.

[tedster]

Questions for Cyclob, sit2510, and anyone else who is seeing this problem --

1. Are you seeing Mozilla Googlebot crawling your pages?
2. Are you fully in the Sitemap program?

I know that you, Cyclob, said that your sitemap was verified -- but are things progressing from their? Are you seeing any statistics?

[asher02]

Same is happening to me. site: returns 11,900 on regular and 255! on bigdaddy. I use sitemaps in full and get statistics. The site is 100% clean. Traffic is down by 50% as many pages lost about 100 spots on serps.

I am clueless...

[Nick0r]

Also for me, huge drop in pages indexed, a lot of my most heavily weighted and link to pages thrown out in SERPS and in site:. Traffic down a bit because of it. Actively use sitemaps, got stats, it appears to work fine.

[Wibfision]

Huge drop in pages indexed in BD for me.

Yes - Mozilla Googlebot crawls the site
No - Not in Google sitemaps program

[Rosalind]

No drastic change in traffic, it's up slightly if anything, but I'm also seeing a reduction in page counts. But before now Google was over-reporting my count, so a website with around 10 000 pages was reported with a site: index of about 25 000.

[Nick0r]

Ouch - all the pages that were removed from my site have gone to Pagerank N/A.

[Armi]

I have a weak indication on the Mot-Bigdaddy-datacenters for months. Only the Mozilla-Bot is very active. On Bigdaddy my big shopping portal has been indexed fully and I have 4 times as much Traffic there. But I have lost URLs strongly on Bigdaddy since Thursday.

[BillyS]

1. Are you seeing Mozilla Googlebot crawling your pages?
2. Are you fully in the Sitemap program?

I know that you, Cyclob, said that your sitemap was verified -- but are things progressing from their? Are you seeing any statistics?

1. - Very active Mozilla Googlebot and GOoglebot

2. - I've been using Sitemap for about 2 months now. (verified and seeing statistics)

I think there is going to be a 2 phase rollout of BD. Infrastructure first, followed by an index update. I rank well on BD and non-BD DCs with the execption I've got roughly 220 pages in BD and 1,030 in non-BD centers.

[Steph_R]

BillyS,

How many visits daily do you get from Mozilla Googlebot? I am curious what is "normal" for one day.

[Dayo_UK]

>>>>I think there is going to be a 2 phase rollout of BD. Infrastructure first, followed by an index update.

I sure hope you are right BillyS

[rookiecrd1]

I am seeing the same thing. Over the last week I have seen pages indexed go from 500,000 to 100,000. My site is prett much an authority site. Some decrease in traffic has been seen due to not having all the search terms in google any longer. I do rank higher for most that are there, but so many pages that were indexed simply are gone.

Not sure what to think.

My other VB forum which is pretty much set up the same has had an increase in pages indexed which is strange

[otech]

All my sites have improved over this last week (one even jumped from 400 pages to 10000 on BD)..

I am still down on what the non-BD DC's are showing, but its heading in the right direction.

I am sure Google will get to all of us guys ;-)

[mycutegoddess]

Could you please specific the IP of that Datacenter.

Today I checked on 64.233.183.99 and 64.233.183.104 the result came up from 11,000 to 142,000 unfortunately, now the result roll back to normal (11,000)

Any suggestions?

[BillyS]

>>How many visits daily do you get from Mozilla Googlebot? I am curious what is "normal" for one day.

For my site, Googlbot is more a hit and run bot. It appears, pulls a lot of pages then goes away for a bit. So I'm not sure what an average day looks like.

Earlier today I had Mozilla Googlebot grab 350 page (35% of the site) over a 3 hour period. After seeing this thread I just checked my logs and Googlebot 2.1 has grabbed about 300 pages over the last hour or so.
Up until today:

Googlebot 2.1 = 3,043 pages
Mozilla Googlebot = 2,559 pages

Despite all this spidering, I continue to lose pages in BD's index. From a high of 1,020 pages I'm now down to 312 (with about 100 of those being supplemental) - an all time low for the last two years.

Came from my post here: [webmasterworld.com...]

[MrRoy]

If I am not using Google Sitemap program, am i facing any sort of penalty for which i noticed that all my internal index pages disappeared from Google.
As far GoogleBot is concerend, it is visiting everyday.
I dunno wat to do and how to react to the present situation.

[classifieds]

Traffic is down about 25% since last Thursday.

I'm seeing a drop of indexed pages on BigDaddy DCs of anywhere from 65% to 80%.

The Mozilla Gbot crawls about 30k pages a day and have been for the last month or so.

And I don't use sitemap.

[dinnerware]

>>>>I think there is going to be a 2 phase rollout of BD. Infrastructure first, followed by an index update.

So... how can we tell when BD starts to do an index update? Thus telling if you have been dropped for good or not?

BD has dropped pages on us it seems every day. No change for us (in SEO actions) than before BD. I have been told by many to sit and wait, but I want to know what to look for.

[t2dman]

Also noticing many pages that have had no. 1 rankings for over a year, Pr5 etc - now lost their cache.

Yes when I check the other datacenters, many of them have current caches. Glad to hear that others have the same issue. Hope its just some bug that Google will be fixing as it makes no sense otherwise.

[sit2510]

Questions from tedster

1. Are you seeing Mozilla Googlebot crawling your pages?

Yes, Mozilla Googlebot does crawl the missing pages as well as the normal bot. In non-BD, the missing pages are still there with latest cache of 23 or 24 Feb 06, but those of the very same pages are not in BD datacenters.

2. Are you fully in the Sitemap program?

No, it is not in the Sitemap program.

I'm not sure whether it is a huge bug in Big Daddy that accidentally lose pages or what else...If so, then everyone is at risk of having Google fumbles its ball with our pages disappear.

[tedster]

BD is not stable, and has a lot of funny data right now. If your pages are getting spidered, then I'd say hold on for a bit before hitting the panic button. You are far from the only one seeing this.

[jo1ene]

Today I checked on 64.233.183.99 and 64.233.183.104 the result came up from 11,000 to 142,000 unfortunately, now the result roll back to normal (11,000)

I am in a similar situation for my one big resource website. I am going to wait it out.

[mycutegoddess]

Are there anyone to point out on [64.233.183.104...] Is it really BD or it's rubbish?

[Nick0r]

In an email from google:

New sites, changes to existing sites, and dead links will all be
noted after our next crawl, which will be completed soon.

Could be the second part of big daddy.

[Dayo_UK]

Intresting Nick0r

One report from SES has Matt Cutts saying 6 weeks for Big Daddy roll out - which is what he said 4 weeks ago.

So how soon is soon for the next crawl completion I wonder.

[Crush]

hmmm. Yes Our main site has 274,000 pages and now it is showing 74,000. Also a lot of well linked pages have gone from the serps. I have seen this before, we came back although it is uncomfotable waiting for it to get back ot normal. Hope this big daddy stuff is implimented sooner rather than later.

[rookiecrd1]

Crush, I am in the same exact situation as you. Numbers are almost exact. Was 290k indexed, now only 72k.

Crossing my fingers. We are an authority site in our niche.

[insight]

Nick0r, above you mention the "second part of BigDaddy" I was under the impression that BigDaddy was a gradual roll-out to the various data centers, not a distinct multi-part update like Jagger. Did you read otherwise somewhere?