Spidering Members Only Pages

Switching Models

[Bogglesworld]

Hi. I have a simple question. I am thinking of switching to a members only/ password protected model. How can I make sure that the googlebot still spiders all of the pages? Thanks for any suggestions that you offer.

[cminblues]

It's an old trick hehe :)
I suggest using Cookies instead, but if you want, voila:

If you have Apache, put this in the VirtualHost section of your 'member-only-access' site:

<Directory /httpdocs/mysite/www/memberarea>
AuthType Basic
AuthName "PUT WHAT YOU WANT HERE"
AuthUserFile /httpdocs/mysitepasswd
require valid-user
satisfy any
order deny,allow
#Googlebot IPs
allow from 64.68.82 64.68.86 216.239.46
deny from all
</Directory>

Of course you must use the 'htpasswd' command to set the uid/pwd in '/httpdocs/mysitepasswd' file.

cminblues

[Slade]

I don't see the point?

Unless you're not going to allow google to cache your content, I can still read your pages through google.

Note, there are more IPs than the ones listed.

[cminblues]

>>Unless you're not going to allow google to cache your content, I can still read your pages through google.<<
Right.
But this is'nt the way a standard, lazy, member-area user, thinks.

>>Note, there are more IPs than the ones listed.<<
Hmmm.. these are C classes, not single IPs.
And I can say that very rarely Googlebot uses IP out of these classes :)

cminblues