How to create Pages visible to search engines but not to users

this is not for unethical SEO, just registration type of site

[followgreg]

Hi Guys,

I have a website with great information but it requires registration before you see the info.

I saw a few sites where search engines can read the whole site but the user is asked to register first.

My site is PHP and I wonder how I can do that, and also if I could be penalized for this?

Can someone help? I am not even sure if I post this under the right forum but...I have no clue what I should do to have this feature :)

[ergophobe]

You can create a white list of user agents (i.e. "googlebot") and let certain ones in without a password. Of course, a savvy user with Firefox could set the user agent to googlebot and get in too, but I don't think that's a large number and if you add IP checking, it's going to get real hard to beat.

Use $_SERVER['HTTP_USER_AGENT'] to get the user agent

As for penalties, I don't know. It would not violate Google's rules regarding cloaking, so I would think it would fine, but I'm no SE expert by a long shot.

[followgreg]

That is what I was afraid of....being penalized, I absolutely don't want to take a chance.

Is there someone to let me know if this is good for penalties?

[grandpa]

Change to this...

it requires registration before you see the info.

...to this...
it requires sessions before you see the info.
...and you describe my websites. A certain list of user agents, including Google, are allowed to view pages without starting a session. There are no SE issues.

You can easily check the user agent and ip range as suggested and let them pass to any page you choose without requiring registration. One caveat is that your pages will be crawled and indexed, but when someone clicks a search result they will be required to register to view the page. That can be irritating to web surfers. You could test for that and direct the visitor to a registration page.

[arran]

Remember to use 'NOARCHIVE' to stop search engines showing cached copies of your pages.

[followgreg]

Thank you guys that will help me, since I'm using Mambo and that I am not used to this solution yet I might find a way to implement your advice :)