Forum Moderators: goodroi
I run a site that authenticates users using PHP. When I look at how my site is indexed via search engines, I obviously have a problem getting the robots to crawl pages requiring a password.
I thought I might have overcame this when I added a button that allowed access to the site 'anonymously'. it submitted a form to my DB with a username/ pw combination automatically. So, I thought a spider would try the button and that would allow indexing of my site. It does not appear to work.
I have read that creation of a site map can help get 'members only' sites indexed by directing how to crawl. But I am having a problem finding examples to show how to do this.
My question is: how do I get a robot to crawl the proper way, and to utilize the 'auto login' button to access and index the protected pages. The pages are protected simply to get users to login for purposes of a rewards system.
Thanks!
