Forum Moderators: open
I use htaccess to protect a directory of my site I only want accessible to members (who enter via a log-in page).
When deep-crawler hits my log-in page it stops, but
my protected area contains keyword-rich and updated content I'd quite like Googlebot to index.
So is there a way of letting the deep-crawler index my protected pages without letting non-members in at the same time?
But as a surfer I would find it to be incredibly annoying that I would get a login page when I tried to go to those pages. I would suspect that Google would frown on this also.
Why would you want protected pages to come up in the search results? Would you allow these pages to be cached?