Google News and News Registration Sites.

[kila_m]

How does the google spider follow the sites where they ask you for registration? Is it done by cloaking for something.. as it seems that whenever you click one of the enws items.. they ask for you details and Email address.. presumably so they can spam you and sell your email address.

Consindering how strict Google Adwords is to advertisers.. Why include these sites in at all? I suspect google is paid for these so they can increase membership.

[Shak]

kila m,

I very much doubt Google is paid for these.

From my experience Spiders such as the google news spider do NOT have to go through the sign-up/log in process, and just goes straight in to read the news.

Naturally this is different for humans surfing using a browser with items such as cookies and session IDs.

I am not a techy, so the above is the best way of explaining it.

Shak

[jpjones]

From a techy point of view, I imagine the scripts on the site look at the IP address and user agent of each request.

If the request for the page comes from a known search engine 'bot (based on IP and Useragent) then the news article is shown. If not, then the site presumes the user is human and so requires the log-in information from the browser (via cookie and/or the user typing in details), and then allows progress as normal.

This allows the sites to get spidered and appear in the indexes of SE's, and still generate revenue.

JP