Big daddy has indexed only 136 of my 20,000 pages

Help Needed: Big Daddy hates me

[darnoc]

Maybe you could help me. my site uses dynamic pages to hold a database of real estate property details. It was clear that google really didn't like the dynamic pages. So, in the hope to improve the number of pages indexed we built a separate set of identical static pages. This worked. Google loved the static version of all pages. However, big daddy seems to hate my static pages. Any ideas would be appreciated.

[lammert]

First of all welcome to WebmasterWorld!

If you have read Matt Cutts' blog, in one of his posts about 302 redirects and Bigdaddy he mentioned:

Matt says:
My only point is that the new infrastructure at the Bigdaddy data center will let us tackle canonicalization, dupes, and redirects in a much better way going forward compared to the current Google infrastructure.

Looking at your site structure of dynamic pages and a set of static pages you specifically created for the search engines it might be that Bigdaddy recognizes this content now as duplicate and that you have been penalized for that.

[zeus]

lammert - penalized is not the case, filtered is the word, if your pages looks a like in content then they will be filtered, ok sometimes they over do it.

[lammert]

According to darnoc only 136 of the original pages are left in Bigdaddy. I am not sure how he tested it, but I assume with the site: command. I would call this a penalty rather than filtering. I agree that the normal behaviour with duplicate content is pushing down (N-1) duplicates so only 1 stays in the SERPs, but since Bigdaddy I am not so sure anymore about it. I have seen two sites in my niche with completely clean content (A wiki and a non-profit organization that have been around sinds the late nineties) that totally disappeared from the SERPs since Bigdaddy. The only reason I can think of is that other sites (legally) copied the content and on the Bigdaddy infrastructure now all versions, including the original one are invisible in the SERPs.

[Phil_Payne]

I think the Web is growing faster than Google can or wants to cope with. The days of having 100,000 pages indexed are gone forever.

Personally, I welcome this. You should have enough scope in 100+ pages to attract attention to what you do, and then provide your own site-level search into the details.

Has the advantage of keeping the searcher on your site, as well, and Google can forget about the 99,900 pages that aren't relevant to most of us and concentrate on the core.

Sitemaps give you a way of pointing the Googlebot in the right direction.

Google is now downloading my sitemap and robots around once a day - it seems to be a 22-hour cycleŽat times.

[otech]

I dont agree with that Phil.

Big Daddy has a huge number of sites that are nowhere near their non-bd levels of pages.

In fact, almost every site I check.

It is completely illogical to suggest that 'the days of getting 100K+ indexed are gone' and that it would benefit anyone.

The beauty of search engines is that you search for 'blue model 455 widgets' and you get a page about 'blue model 455 widgets'. You suggest it would be better to just land on any old site mentioning 'widgets' and then hunt around that site for specifics.

Futher, that would also just mean that we would have to register many many domains instead of just working and building 1 quality site - again this is completely against everything thats come out of google.

Compare a large number of sites against the old infrastructure and I beleive you will find the BD index is not anywhere as complete as the older one, and this topic has been discussed in more depth here: [webmasterworld.com...]

[arbitrary]

Compare a large number of sites against the old infrastructure and I beleive you will find the BD index is not anywhere as complete as the older one...

This may be a very valid observation but what some are suggesting here is that pages are being kept out of BigDaddy because they are deemed to be dupe pages. Therefore a smaller index. Not sure if I agree this is happening, just want to restate what I think is being said.

[Phil_Payne]

> I dont agree with that Phil.

> Big Daddy has a huge number of sites that are nowhere near their non-bd levels of pages.

> In fact, almost every site I check.

Isn't that what I said?

Why should Google host 100,000+ pages of someone's catalogue at no charge? Those days are gone - stop looking at it as a problem because it's not going to go away. It's now a situation.

You will no longer be able to get every nut and bolt indexed - you have to make priority decisions.