Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google News doesn't want to crawl commentpages

How to remove commentpages from crawling by Google News and not by GoogleBot?

         

Tiffny

9:44 am on Jun 18, 2007 (gmt 0)

10+ Year Member



We've had some emailcontact with Google News to be included in their service. Now we're almost there, except they don't want to crawl pages when comments are displayed. They suggested we edit our robots.txt to disallow googlebot from crawling the article-pages with comments. Since we only show the full articles with the comments beneath it, you can see our problem. How do we show Google News the pages WITHOUT comments and the normal Google Bot the page WITH comments? I read on this wonderful forum there's no such thing as a Google Newsbot, only the Googlebot. Do we make a special "permlink" to pages with the full article without comments? But how would that affect our crawling of the commentpages?

How would we go and pursue this? I already searched everywhere, but never found a mentioning of Google News not wanting to crawl pages with comments.

tedster

4:31 pm on Jun 20, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hello Tiffny, and welcome to the forums.

This is a new one to me - I see plenty of pages in google News results that include comments. Sometimes they're hidden until you click on a "Comments" link, but they're still in the source code.

I don't work with any sites that are in Google News at present. I hope someone else comes along with a helpful idea for you.