News site's posts now take a week to make it to organic SERP

[devil_dog]

I run a news website posting hundreds of news articles daily. Have been included in Google News for almost 3 years now.

Since about one month or so, our new posts are taking exactly (to the second) 7 days to show up in regular search.

The new posts are immediately pucked up in google news and frequently featured on the news cluster on organic SERPs for related keywords.

On doing a search for full title or a chunk of text from a news article (published within last 7 days), we see only scraper sites like ones who copy the title or a snippet of text and link back. sometimes even tweets show up. My site is nowhere in sight. not even with "&filter=0" . However, the link from my site is present when choosing "latest" option on Google.

One strange thing is when doing a site search with query say:-
site:example.com {story title or chink of text}
the link from my site is rightfully there.

We haven't noticed much change in Google Traffic cause older stories now are strangely performing better in organic SERP. (Google still seems to value internal links from new posts).

Anyone fill me in as to why this may be happening? has anyone faced it before?

[edited by: Robert_Charlton at 7:06 pm (utc) on Jan. 3, 2010]
[edit reason] use example.com - it can never be owned [/edit]

[devil_dog]

@BedSupperclub for my personal blog ( less than 1 post/month) i dont cut articles and dont have problems. however i think its a good practice to cut articles... atleast from category, tag, archives, etc if not homepage

For the site thats having this issue, i show only first 35 words...

[BedSupperclub]

well cutting my articles didn't help...

Why is google bugging us with that 7 day purgatory? if our sites were that bad, then there would be no reason to reintegrate them after 7 days.

i wish bing to get 50% market share so goog will stop behaving like this!

[devil_dog]

just wanted to update. since today my new stories have started indexing again.

On a related topic i also noticed an interesting interview of Matt Cutts which details issues regarding indexability. Webmasterworld discussion at [webmasterworld.com...]

Gonna take some time off to study the transcript. a nice summary at SeoMoz [seomoz.org...]

[devil_dog]

Just an update to anyone whose been tracking this issue.

One week after my above post on march 17, the problem started again until few moments ago. Here is how i may have fixed it...

about 3 days ago i was going thru much much older sections of the site... Noticed i had many duplicate posts... in some cases 20+ copies of the same article. (how it got there is not the point of discussion here).

I deleted the duplicates, 301'd them to one copy (choosing by doing site search) ,,, probably deleted about 1500+ (out of 350000) posts in the process.

When I was done, google webmaster tools magically showed me 0 urls indexed for all sitemaps. I paniced, but calmed down when others on webmasterworld noticed the same behaviour. At the same time, Googlebot fell in love with my site and started crawling hard and deep. requesting 1 to 5+ requests/second (i set 10/sec in settings in GWT).

A day or more after banging my server, GWT showed almost 100% indexing even on old sitemaps which had < 50% indexed urls in GWT. (for posts i have 7 sitemaps containing 40k - 50k URLs each).

Also 410'D a section of my site which is not linked to and not used for ages. The crappy script used to run it made loads and loads of list pages with nothing in it but still returning 200 status. - doing site: + inurl: shows almost 50k results containing nothing meaningful to users nor bots.

So moral of the story is get your site in order, reduce link and crawl budget leakage .... that or Google fixed something at their end.