Forum Moderators: open

Message Too Old, No Replies

Can Googlebot deep-crawl htaccess protected directories?

I want my protected area to be crawled

         

soccer_star

2:19 am on Dec 2, 2002 (gmt 0)

10+ Year Member



Hi,

I use htaccess to protect a directory of my site I only want accessible to members (who enter via a log-in page).

When deep-crawler hits my log-in page it stops, but
my protected area contains keyword-rich and updated content I'd quite like Googlebot to index.

So is there a way of letting the deep-crawler index my protected pages without letting non-members in at the same time?

BigDave

2:45 am on Dec 2, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes there is.

But as a surfer I would find it to be incredibly annoying that I would get a login page when I tried to go to those pages. I would suspect that Google would frown on this also.

Why would you want protected pages to come up in the search results? Would you allow these pages to be cached?

jomaxx

6:14 am on Dec 2, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I totally agree that password-protected pages are a PITA to deal with and should not be in Google. I would personally call it a form of cloaking, but it's possible that Google has an explicit policy covering this situation in one FAQ or another.