Google has stopped indexing new posts and articles

[steerpikegg]

Hi, I was wondering if someone had any suggestions as to why this may be...

First some background info - the site is a retail and consumer information site and has been established and ranking well for the last 4 years.

Up until around 2 months ago, when I added new articles or news item, Google would have them indexed the same if not next day at latest.

Since then, if I do a site:www.widgets.co.uk and look for the new articles or do a 'Past 24 Hours' filter, nothing shows up. In fact hardly anything has shown as being new in the last few weeks.

Googlebot has been hammering my site like crazy (it craweled over 185,000 pages on Wednesday), but just seems to ignore the latest pages.

Any suggestions are more than welcome.

[tedster]

I'd first suggest that the site: operator results may be buggy - lots of members are seeing this. How is traffic to those new pages? Is Google sending any? If so, then the page is indexed, no matter what the site: operator says.

[FranticFish]

The site: operator is screwed beyond belief right now. Just when you think it can't get worse...

I was looking at a new site yesterday. site:domain.com showed 400 pages. I know the site had thousands. By querying different folders and dynamic pages I turned up over 15,000 urls!

With all the extra stuff Google have added to WMT lately, this basic query going to the dogs is so annoying. It's such a a basic diagnostic tool, wearing a whiter than white hat - "what does Google know about my site?"

[steerpikegg]

Hi Guys,

Thanks for the replies, however even if I do a plain search for exact title match and when I check statistics, no traffic for these new, highly topical articles is coming via Google - it just doesn't seem to make any sense.

If I run the same queries on competitor sites, I see that they are having large numbers of new pages indexed in the 'last 24 hours' filter.

My home page is kept fairly up to date, and this includes links to the new content, but it seems that Google won't index the content associated with these links.

Nothing drastic has happened to my site in terms of changes or SEO, and I do not have anything of dubious or black hat nature.

I guess I can only hope that the situation resolves itself.

As a side note, one of my main keywords which has been page 1 top 3 for the last 4 years has been bouncing around from page 1 to page 20+, so I don't know if this may be related.

[tedster]

no traffic

That's the telling sign. First I suggest checking the basics:

1. robots.txt is still as intended (you'd be surprised!)
2. check your new pages for an incorrect robots meta tag
3. check your server logs to see if googlebot is calling and how your server responds
4. check Webmaster Tools for crawling data, messages, etc.

[steerpikegg]

Hmm, I've checked all the obvious, robots is still allow from all, no noindex meta tags, no canonical issues.

Googlebot is hammering the site continually, and the articles are being crawled, just not put into the index. The thing is, these are topical articles which are relevant only for a few days to a week, so if they don't get into the index in a reasonable time frame, there's not much point.

Maybe it's something to do with the caffeine weirdness going on, but it make me wonder why it seems to affect my site and not competitors.

[FranticFish]

Is your link profile markedly different from your competitors?

Have you tried 'fetch as Googlebot' in WMT on a few different pages?