Pages being dropped from Google's index?

[erpa1119]

We have an ecommerce site with about 109,000 products.
45,000 are currently active, the others are currently out of stock,
and may or may not come back in stock ever.

In January a site:www.example.com showed about 100,000 urls.

Today we are showing less than 40,000, and appear to be dropping about
10,000 a week.

I have 4 sitemaps in google webmaster tools with ALL product links and
some category pages plus the homepage.

I have a decent robots.txt file and have recently (just yesterday)
requested a removal request for a subdomain that we are integrating
into the product pages(reviews section)

I am concerned that I have induced a penalty.

The site is as clean as I can get it and I actively monitor my stats
etc, including disallowing known bad spam bots so that they do not try
and steal my content, to post on their site.

Any advise or thoughts are appreciated.

[edited by: tedster at 12:10 am (utc) on Mar. 30, 2008]
[edit reason] switch to example.com - it can never be owned [/edit]

[ambellina]

Are these pages being refreshed with new/different content on a regular basis?

Are they easy for Googlebot to find in your navigation structure?

If either of those are a no, then G is probably assuming that a lot of those pages are dead or unimportant and it's either dropping them completely or just not visiting very often.

Add some new content and link to those pages from some of your more popular ones and see what happens.

[tedster]

It's interesting that the numbers line up pretty closely between your in-stock items and urls indexed. The numbers in the site: operator search have been quite volatile recently, and it's pretty dicey to draw conclusions from that data alone. Still, the conicidence is "very interesting".

If your 40,000 urls are still drawing Google traffic, I don't think you've got a penalty. You're just caught up in the current flux that's going on.

[erpa1119]

tedster,
Is it possible that google knows if our items are in stock or not? We do have ALL of our active items in google base as well which gives us ~15% of our traffic.

I need to read up more on the current flux that is going on, can you point me to a good thread about it? Ill look on my own but if you do, please reply with the link.

Thanks

[tedster]

We run a thread every month for watching the Google SERPs changes - this month's thread is here [webmasterworld.com].

You'll notice that just around the Easter holiday the thread picked up in activity, with posts like "my site disappeared as well, after being #1 off and on for over four years", "In my case it's not traffic, the sites gone" and so on. Other threads here contain similar side discussions over the past week.

What exactly is shaking out is not yet clear, but most experienced webmasters will tell you not to freak out at times like these, and definitely not to make any major changes based on what you see until things stablilize a bit.

You're reporting on the number of urls in the index - but how is your search traffic and the conversions you get from that traffic? That's where the essential stats are.

[erpa1119]

Search traffic is off by about 12% or so from December levels, conversions however have decreased by about 25%, which I am attributing to the overall economy.

Side note, I updated my site-maps in Google/webmaster console today, which was long overdue.

[erpa1119]

Good news tho, Yahoo traffic and bot's have picked up at an ever increasing level.

[tedster]

So the site: operator shows a decrease of 60%, but your search traffic right now is down about 12%. That says something more about this shift for you. In essence, the "damage" is a lot less than you migh have first expected, just from looking at the site: operator results.

Now you can probably get much fuller insight if you analyze for which specific search terms have declined the most. Even after you identify those key terms, I'd still suggest waiting another week or so before undertaking any repair work on their associated organic landing pages.

You might also get some further insight by reading this recent thread [webmasterworld.com]. It's very possible that the number of urls you actually have indexed in Google is higher than a basic site:example.com operator shows!

[erpa1119]

Just ran a log report (weblog expert):
Uniques are up by a factor of 2.

December to Yesterdays range (uniques) was
1000 - 1800

Perfmon Report:
Max "Sessions Current": 687

WebLog Expert Report Range: 3/28/2008 19:00:02 - 3/29/2008 22:33:04

Total Visitors 9,065
Total Unique IPs 3,099

Bot Report:
1 Yahoo! Slurp 9,855
2 MSN Robot 6,974
3 Googlebot 2,281

[erpa1119]

forgot this:

Unique Key phrases in report:
Google: 624
Yahoo: 10
AOL: 2

Need to work on Yahoo and MSN, any advice?

[tedster]

We've got other forums here that are dedicated to those two search engines - full of good stuff plus the occasional rant ;)

If you can access reports from December and then compare the specific keyphrases tothe current report, you'll be closer to the information you can act on.

[nolimitdomains]

Interesting to say the least.
My site URL lost over 5,000 listings in just a month.
Funny thing is:
I wasnt getting much traffic anyways,
But, I was hoping to build my links up steadily.
Then Google struck! lol..
Im getting the impression that this is happening to everyone right now..?

[minnapple]

Think about what makes a page "worthy" of being included in the search index.

If you have a product with several variations, do you think each of these products are worthy of being included in the index?

Google will only list two of these in any search at best, so what is point to have the others in the search index?

Google has the bulk of the data for each page, but only releases a certain amount to the search index.

Google can change the parameters on how much data to release to search index by changing link weight and content parameters etc . .

[nolimitdomains]

Okay then.
How about you just list one link to your site..
on each major search engine..
and hope for the best.
Let me know how that works for you!

Ill be building as many as I can over here
because I am not independantly wealthy and I need
as much exposure as I can possibly get.
Have a great evening!
Joey A.

[Robert Charlton]

Interesting to say the least.
My site URL lost over 5,000 listings in just a month.
Funny thing is:
I wasnt getting much traffic anyways,
But, I was hoping to build my links up steadily.

IMO, this approach is backwards. You should be increasing the number of pages as your inbound links will support them, not hoping your links will catch up before your pages fall out.

I see an awful lot of posts where people are lamenting losing thousands of pages, and you can read between the lines that their inbound links and their PageRank (yes, PageRank) is not all that great.

I feel that a lot of the reports of sites being "penalized" when a great many pages are added has much more to do with inadequate link juice and badly conceived PageRank distribution than it does with the wrath of Google. Ditto with pages being dropped.

[nolimitdomains]

(yes, PageRank)
Most of us have no PageRank.
Its no secret.
The only way to build PageRank is what my friend?
hmmm....
Back to square one.

My site is generic and done well.
There is plenty of information there..
I just cant change it.

What Im shreiking at is those people:
Who pretend to create new content on their sites
when actually they're just mixing the same pages up...
day after day.. after day.

Gee, new Meta Tags again(on every page)?
A new Page Title again(on every page)?
Give me a Break Man.

That isnt Page Juice, thats Page Hustle.

[Quadrille]

"The only way to build PageRank is what my friend?"

incoming links.

Don't worry too much about quantity, benefit tends to come form Quality Links.

Here's the math:

(Site + Quality Links) = (Site + mostly useless links) - (unrelated reciprocals + useless links)

[edited by: Quadrille at 12:33 pm (utc) on April 2, 2008]

[nolimitdomains]

ah...
"Quality links"
Every link has the potential to reach others...
as long as it isnt Immoral, Illegal, or Placed on Planet Pluto.
All three of which I have zero interest in.

Yes, I actually mentioned morals on a search engine forum.
Imagine that.
lol..