Google News does not crawl website news anymore?

[DefenseTalk]

Hello,

It seems that google has stopped crawling our website's news section on july 20. What makes that so unusual is that google news bot has been working fine on the site before that and our news would appear on google news within minutes after publishing. We did not change anything on our side and the investigation I have done from my side has revealed nothing that would prevent google from crawling it.

Google does come on the site, it crawls all other sections and even sometimes goes into the news section as well...

I sent couple of emails to Google News Team but it seems my emails just get lost somewhere.

Is anyone else having this issue? How can we fix it? What is causing this problem? If anyone could answer these questions...

Thank you much.

[vincevincevince]

Have you not had live news recently? Has your news been abused for spam, etc? Finally, have you changed your site or page design?

[DefenseTalk]

We add news daily up to 8-16 items... I don't know if it was abused, how do I go about checking if it was? Page design was changed over two months ago but it was spidered without any problem until the 19th.

Thank You.

[kamikaze Optimizer]

Hi DefenseTalk:

You are not alone. Odd things have been going on with the Google news bots lately.

They stopped coming to our site also, but I wrote to them and they came back again, this has happened twice in the past month.

Also, the team that works on this is not in the USA, but in a country that I block all IP's from, which made it all real confusing on my end.

I would say, just keep emailing.

[DefenseTalk]

Hi Kamikaze,

Thanks. I have my fingers crossed that it's something similar to that which I think it is. Recent changes to the google algo has rattled some cages but I am willing to wait as long as it comes back otherwise we will have to look for other avenues to get the visitors.

Thanks again.

[nutsandbolts]

Same here but it's been 3 months for me. Every time I e-mail they say it's "a technical problem, please be patient as we are working on it" - but despite every other news/blog service able to crawl me, the Google News bot cannot anymore it seems. It's probably an editorial decision, although they have never said that to me...

Hundreds of sources have been dropped over the past 3-4 months from the Google News section, and the whole ranking system has totally changed in the past few weeks - with only major sites getting any visibility.

Shame really...

[kamikaze Optimizer]

nutsandbolts:

I have had editorial issues in the past and Google had no problem telling me, so I not sure thats what your problem is.

Also, I am far from being a major news source but I do very well with Google news.

[DefenseTalk]

Does anyone know why would be google hitting .shtml pages in our news section when everything is .php or .html? It should be hitting the .php pages not the .shtml... any ideas?

thanks

[vincevincevince]

I believe Google manually sets you up with what it recognises as a 'news' URL. Did you previously have news using .shtml?

[DefenseTalk]

yeah, before it was .shtml page and then changed to .php but it has been crawling .php pages as well. Iin fact it automatically started crawling .php pages until july 20th... when it wouldn't even pick up any news, etc. I did clean up the news page (removed RSS feeds from the forum) and emailed google news back, lets see what they say now.

thanks

[vincevincevince]

You may be able to use mod_rewrite to rewrite .shtml to .php - not the solution you are looking for I guess, but it might help?

[DefenseTalk]

I just need something to redirect the user to the .php extension. All file names are same, path is same, just the file extension has changed.

How can it be done?

thanks

[cellularnews]

I had a similar issue to this last year.

At about October time, we started buying in additional news articles and disapeared from Google News overnight.

It took a LOT (an awful lot) of testing, emailing and begging to get absolutly nowhere with a solution.

Then - an accident occured.

We had gone from an average of 8 articles per day to 18 per day - and we had links to 2 days worth of news on the front page - so a jump from an average of 16 links to 36 links.

One day, we ended up by accident only showing that days worth of news - and that very day reappeared in Google News.

Fixed the bug, and vanished back out again.

So - we had a clue, and more testing showed that the number of links on the news (or front) page for some reason kills Google News.

We now edit our front page to never start the morning with more than 12 articles (that morning, plus a few from the previous day) in addition to other navigation links and everyday we are in Google News. We see a drop off in the evening when the number of articles that day exceeds 19 headline links.

To further verify, I checked some other sites with loads of links, and they also rarely appear in Google News.

I have left well alone since then, but looking at other sites, leads me to suspect that it is not just the number of links on the front page, but the ratio of links to plain text that matters - a site with just headline links and little else suffers quicker than a site with lots of links and maybe a paragraph of text per link.

We are slowly increasing the quantity of plain text on the frontpage per news headline and testing with additional news links to see if that holds water.

[vincevincevince]

I just need something to redirect the user to the .php extension. All file names are same, path is same, just the file extension has changed.

Either using mod_rewrite, or you can bypass the problem by going back to .shtml and setting php to parse the .shtml file extensions.