Forum Moderators: open
do crawlers stop when they come across a login page? can anyone answer that
thanks
Available only after logging in then the crawlers wont crawl through rather wont even reach there?
That is correct.
There's a reason why that content is behind a login, correct? I mean, you wouldn't want that available to the public would you?
One such method (being discussed in a parallel thread in this forum) would be to redirect all useragents except search engine bots to the log-in page...
Site maps / Crawling
[webmasterworld.com...]
There are also ways of setting up cookies to redirect all useragents except bots (which don't execute scripts and don't read cookies) back to the log-in page.
Arguably, these methods might be violating the terms of services of various major engines, and your site might be penalized for using these methods.
There's also some ambiguity in this area, and perhaps something of an uneven playing field, as some major sites that the engines want to have in their indexes have been permitted to use cloaking to get log-in protected content indexed. Generally this is done with the full knowledge of the engines.
My experience with sites that are indexed and use log-ins is that they lose over half of their search traffic at the log-in point. If your site is well enough known, that percentage may well be less and that loss may not matter. If you're a new and unknown site, that loss of traffic might be a big factor.
