GoogleBot and Restricted pages

[gollyme]

I accidentally found a restricted page or document via Google. I dont intend to get around the subscription but I managed to save $50 and the risk of getting my card hacked.

And now Im just wondering how we can ask GoogleBot to exclude documents in the restricted folder. How we can make sure that documents "restricted for members" remain restricted.

Thanks,
GollyMe

[jatar_k]

Welcome to WebmasterWorld gollyme

Having a login form that runs through a login script would help. No links into the restricted are that don't require you to login first. Having an authentication script that makes sure anyone trying to view the page is authorized.

A robots.txt that disallows all protected content.

Those will help.

[edited by: jatar_k at 9:40 pm (utc) on Jan. 12, 2003]

[gollyme]

Hi Jatar,

But how did GoogleBot found the document in the other site with login and other security implements. It bothers me to death.. :o

(ps, it's not just a nickname *hehehe*)

[jatar_k]

If someone somewhere has linked to one of those documents then the bot will find it. If the doc is then served to the bot it will index it.

You have to protect those things that you don't want indexed on your side. It's not the fault of the bot that protected content isn't protected.

[gollyme]

" has linked to one of those documents ". It is just a link not a copy. A virtual folder holding all restricted docs must not be visited by the bots. How can it get there?

I will look for FAQs on how to protect pages from bots.

tia,

[Key_Master]

Robots.txt will prevent Googlebot from following the link but will not prevent Google from listing the link in search results, even if the link points to a 401 protected file or directory. In other words, you have to hide restricted links from all bots.

[gollyme]

In another post that I saw, one guy suggested on using code to hide the docs say php?code=123 but I commented that maybe the bots can somehow get around it and login "anonymously" which is impossible of course.