Subscription required?

Looks like a Cloaked Feed for News Stories?

[Brett_Tabke]

I just ran into a link at the Google.com News Links [news.google.com] page that stated "subscription required".

How'd the link get into Google in the first place? The only way I can imagine is via a cloaked feed or hand edited.

[cminblues]

I've seen this many times also on regular serps, in mixed info-scientific/commercial areas.

[Slade]

I've seen this as well, in some research for SQL information(regular serps).

[Brett_Tabke]

Regular serps are no problem - they run into links to the pages and list it based on the linking text. This is in news where there is only one or two links on the entire web to the articles at that time. They are fully indexed stories - I think they spidered the destination page.

[cminblues]

they run into links to the pages and list it based on the linking text.

Not only..
In some of my examples, the "Forbidden" pages were cached on Google.
Some others, no page cached and no "..terms contained only in link pointing to..".

cminblues

[chiyo]

Google spiders our news index page on one of our sites. Havent researched the actual robot name but it seems to be once an hour. It seems intelligent enough to realise what our new pages are and ones which have just been modified, and also to know which are navigation links/ad links and which are real new news items. On balance they may be more likely to be reading our RSS news feed - that would account for their "intelligence" in these respects. (see also last para below)

Brett is correct i would think. Most news sites provide a small summary somewhere on the news page, or adjacent to the link, or in the description field of a standard RSS-XML feed, even if the full article is only available to registered or subscribed users.

google at least lets you know whether a "subscription" is required. Moreover's feeds have degraded badly for at least a year as (among several other reasons) the actual link doesn't usually indicate whther the article if "open" or requires registration/subscription, (As well as the fact they fill up a lot of their categories with news feeds that pay for exposure - for example the "management" feed is full of news-cum-press releases from a few leading management consulting companies - to te extent that almost 70% of items come from there. (Google at the moment has a great competitive advantage over Moreover for these two reasons)

Google found our news service by itself. It was neither suggested to them (at least by us) nor do we have to do anything than just keep publishing news!. That may not be the case with everyone but it is the case with us.

Apart from just knowing what our news index page is and spidering it for new links, the other possible way they are finding new items though our RSS news feed or services like newsisfree, magportal and other news aggregators

[Brett_Tabke]

If you look at the links there, they are matched with other stories that are the same story (multiple ap stories). So, somehow, they figured out what the story was and matched it with the other duplicates. eg: they spidered and indexed the final page and then put "subscription required" next to the link. Looks like a "trusted feed" to me. Which leads to the next logical que$tion...

[Slade]

I know you've classified and identified a particular breed of serp, but in the case I was referring to, I could choose Cached and read the page, even though I couldn't just click through and read it.

[xplorer]

It's appalling to see some of the biggest sites stoop so low. But this is nothing new. The internet is still beginning. Expect lots of dirty tricks to be used for the forseen future.

One of the biggest gaming sites on the internet has been using the hide from users and show to search engines technique for some years.
The list goes on...

[Brett_Tabke]

I think you misunderstood what I was suggesting. I was considering the possibility that this was ok'd by Google.

[bird]

If you and I can subscribe to the New York Times site for free, why should Googlebot be unable to do the same?

[Brett_Tabke]

Because GoogleBot does not accept and resend cookies like you or I can.

[cminblues]

Because GoogleBot does not accept and resend cookies like you or I can.

Cookies management?
IMHO not so difficult for Googlebot, thinking of a limited news-sites area.
After all it's only a few bytes header's line, in both reading & resending.. :)

cminblues

[danny]

Aaarrgh. If they're going to add subscription - even free registration subscription - pages to their results, they should have an option to exclude them. Or better still, make that the default and have an option to include them...