I have a few links on a new site which require a log-in. Are these in any way troublesome to bots? What happens when spiders/crawlers/bots come across links they can not follow?
Shakil
8:11 pm on Jan 20, 2003 (gmt 0)
Jon,
is it NOT situations such as these where Cloaking comes into play?
Shak
hakre
10:20 am on Jan 22, 2003 (gmt 0)
shakil, excuse, but what is cloaking. in know the meaning of the word but not the meaning in your post.
starec
10:55 am on Jan 22, 2003 (gmt 0)
Cloaking is serving different content to different users (differentiation by IP, user agent etc).
Bots can't log-in. Still, those restricted-access pages can be served to Googlebot, while keeping the login for human visitors.
It is often done by news sites and other pay-for-content sites.
hakre
11:05 am on Jan 22, 2003 (gmt 0)
ah, ok. then my first idea matched it exactly. so you can add the google-spider useragentstring in your browser and surf the cool way over these site - that's nice.