homepage Welcome to WebmasterWorld Guest from 67.202.56.112
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
Spidering Members Only Pages
Switching Models
Bogglesworld




msg:194908
 4:18 am on Oct 27, 2002 (gmt 0)

Hi. I have a simple question. I am thinking of switching to a members only/ password protected model. How can I make sure that the googlebot still spiders all of the pages? Thanks for any suggestions that you offer.

 

cminblues




msg:194909
 4:56 am on Oct 27, 2002 (gmt 0)

It's an old trick hehe :)
I suggest using Cookies instead, but if you want, voila:

If you have Apache, put this in the VirtualHost section of your 'member-only-access' site:

<Directory /httpdocs/mysite/www/memberarea>
AuthType Basic
AuthName "PUT WHAT YOU WANT HERE"
AuthUserFile /httpdocs/mysitepasswd
require valid-user
satisfy any
order deny,allow
#Googlebot IPs
allow from 64.68.82 64.68.86 216.239.46
deny from all
</Directory>

Of course you must use the 'htpasswd' command to set the uid/pwd in '/httpdocs/mysitepasswd' file.

cminblues

Slade




msg:194910
 6:11 am on Oct 27, 2002 (gmt 0)

I don't see the point?

Unless you're not going to allow google to cache your content, I can still read your pages through google.

Note, there are more IPs than the ones listed.

cminblues




msg:194911
 6:39 am on Oct 27, 2002 (gmt 0)

>>Unless you're not going to allow google to cache your content, I can still read your pages through google.<<
Right.
But this is'nt the way a standard, lazy, member-area user, thinks.

>>Note, there are more IPs than the ones listed.<<
Hmmm.. these are C classes, not single IPs.
And I can say that very rarely Googlebot uses IP out of these classes :)

cminblues

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved