Forum Moderators: open

Message Too Old, No Replies

Password protected sites

Are they crawled

         

Northernladuk

2:10 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



I haev a site which has about 10 pages viewable to everyone via footer and header includes.

From here you have to register to use the advanced features of the site. If you try and access one of the member pages directly from the browser you get taken to a log in required screen.

Will google index the members pages by looking at the code or will it be re-directed by the code in the site if you get my meaning? Is there any way around this or is it normal not to want your membership pages spidered?

Many thanks in anticpation

abates

9:12 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



Googlebot can't fill in forms and register on sites.

I've always felt requiring membership to a site in order just to look at it is contrary to the open nature of the web, but that's another discussion. :)

HenryUK

11:27 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



I'm sure you have good reasons for requiring registration. However, those pages aren't going to be indexed. If they were, then people could look at the caches of the pages without registering - which might upset some people in your position.

some sites use a "tease" method - allow some parts or a summary of a page to be viewed by spiders/non-registered users and then you have to register for more.

I think this is probably the closest you can get to having your cake and eating it!

;-)

Northernladuk

12:30 am on Mar 4, 2004 (gmt 0)

10+ Year Member



Thank you very much for your replies. What a top site this is...

A very interesting debate indeed Abates.. It's a dating site so that about wraps the discussion on my part but as you say generally a good topic to banter about :D

kaled

12:37 am on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



In theory, you could use cloaking to let Googlebot in. You would also need to place a NOCACHE robots meta in the header of each page. I wouldn't like to hazard a guess as to what Google's policy is in this area - probably they don't have one.

Having said that, I don't think there's an easy way to get any PR for private pages so you might be wasting your time. However, if you have some high PR sites, you might be able to work something out.

Kaled.

BigDave

2:27 am on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Would you really want to upset searchers that much?

When you are trying to find information about something, do you really want to have a registration screen pop up when you click through from a search engine?

When I find that sort of thing cluttering up the SERPs, the first thing I do is report it to google. I don't know if they pay any attention to my reports, but those sites don't tend to last long in the SERPs.

abates

2:57 am on Mar 4, 2004 (gmt 0)

10+ Year Member



Yeah, if I did that, I'd be worried about Google (or any other search engine for that matter) spotting it and banning me.

trimmer80

3:56 am on Mar 4, 2004 (gmt 0)

10+ Year Member



what if.... on the login page he put a noscript tag at the bottom, with a link into the registered section, providing the registered pages didn't have auth system in them.

I guess it depends on the business model of the site. If registration is not critical (just a method to harvist user details) then the above may be a solution.

a1call

4:08 am on Mar 4, 2004 (gmt 0)

10+ Year Member



Hi,
I just saw a story from la times on google news about the mars rovers. Clicked on it and it said you need to register or login to se the story. So I think It depends on your pages importance.

a1call

4:23 am on Mar 4, 2004 (gmt 0)

10+ Year Member



Hi again,
I thought about it and unless google is a registered user of la times, it is probably cloaking. They probably don't require login when your user agent has the word googlbot in it.

g1smd

11:57 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



That is easily confirmed if someone here temporarily modifies the UA string of their Browser (Mozilla allows that) and visits the site.

dannyboy

11:51 pm on Mar 5, 2004 (gmt 0)

10+ Year Member



This site and its forums are indexed by google, but are password-restricted. There's no cache link next to the listing. If I did a google search and I clicked on the webmasterworld.com search results, I would be prompted to login.

kaled

3:22 am on Mar 6, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Anyone can view pages on this site - you are simply logged as user Guest. Presumably, therefore, Googlebot can read this site as a guest.

I would not be surprised if cloaking were used but I don't see that it is strictly necessary in this instance.

Kaled.

dannyboy

6:01 am on Mar 6, 2004 (gmt 0)

10+ Year Member



kaled, perhaps some forums are accessible in guest mode, but certainly not this google forum.

Try this:

Log out and delete your webmasterworld cookie. Go to google and search for "florida google update"

A webmasterworld forum thread is search result #7 and #8.

When you click either link the thread isn't displayed. Instead is the following message:

"status: Either we require login from users from your ISP because of abuse, or the thread is marked members only. Please login and then back up to view."

I doubt my ISP is marked as abusive, because that would be a big chunk of the internet market.

The google bot is indexing these forums while general surfers can't access it directly unless logged in. Even though google doesn't provide a "cache" link for any of these forums, if you go back to the search example I gave you can see that the search results do display some content contained within the page as its link description (i.e. ... Google Update History. In February of this year, WebmasterWorld began naming the
updates under the same naming scheme as the US ... Florida, 2003, Nov, 14, Fri, 6:37am, ...)

kaled

11:36 am on Mar 6, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



DannyBoy,

I stand corrected. I just connected as Guest and found only three threads in Google News apparently updated today. However, when I logged in there were about a dozen threads updated today.

I guess some threads are cleared for public reading but not others.

Kaled.