Hi. I have a simple question. I am thinking of switching to a members only/ password protected model. How can I make sure that the googlebot still spiders all of the pages? Thanks for any suggestions that you offer.
It's an old trick hehe :) I suggest using Cookies instead, but if you want, voila:
If you have Apache, put this in the VirtualHost section of your 'member-only-access' site:
<Directory /httpdocs/mysite/www/memberarea> AuthType Basic AuthName "PUT WHAT YOU WANT HERE" AuthUserFile /httpdocs/mysitepasswd require valid-user satisfy any order deny,allow #Googlebot IPs allow from 64.68.82 64.68.86 216.239.46 deny from all </Directory>
Of course you must use the 'htpasswd' command to set the uid/pwd in '/httpdocs/mysitepasswd' file.