Can spiders crawl members only pages?

[AjiNIMC]

Hi,

I have some pages with good contents but that are basically meant for members only, that means there are no straight links to the members only pages.

Will spiders crawl those pages?

Thanks

Aji

[Dreamquick]

Unlikely unless a spider got lead there by the toolbar, otherwise it wouldn't be able to discover a way into the members area and so couldn't spider it.

Also bear in mind that if you employ authentication features etc. then that will stop a crawler dead when it encounters a page which has such a feature enabled.

- Tony

[sun818]

I see web sites that are indexed in the search engines. But when you click on the link, you are taken to a sign-up page. There's some sort of cloaking going on where the web sites gives the crawler "free membership" ;) and is allowed to to crawl the entire web site.

grammar edit

[edited by: sun818 at 9:23 pm (utc) on Oct. 15, 2003]

[BlueSky]

I don't know if that's really considered cloaking. The pages exist, but the access requires the person to get an account. Who knows some may not be intentionally letting bots in though I bet many are.

There are some offline site capture tools which claim they can download an entire site to include password protected pages. So, you might want to build an .htaccess file containing at least those.

[sun818]

What would you consider it then? Here's how I understand cloaking [searchengineworld.com]:

Cloaking is the process of delivering one version of a page to a user, and a different version to another user such as a search engine.

Is there a distinction I'm missing? Cloaking is not always a dirty word.