homepage Welcome to WebmasterWorld Guest from 54.237.98.229
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
Forum Library, Charter, Moderators: coopster & jatar k

PHP Server Side Scripting Forum

    
Indexing restricted pages in php forum
aff developer

5+ Year Member



 
Msg#: 3140406 posted 12:11 am on Oct 31, 2006 (gmt 0)

Hi guys

I had a quick question and couldn’t find it anywhere on the forum and was wondering if anyone would know please?

I am just wondering how you get all your forum pages indexed (like in webmasterworld.com) in the search engines without having the cache pages viewable as well? Also how the googlebot gets into the forum without having a cookie set? I would be keen to do this to my small forum that I keep as a paid only forum and I want to respect the members by not making all the information within it public, but still being able to generate traffic from search engines. Any help would be greatly appreciated!

Thanks in advanced

 

mcibor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3140406 posted 9:16 pm on Oct 31, 2006 (gmt 0)

As I remember, the spiders only index non restricted pages.
However if you want to give them access to your more privat stuff, then you would have to check if the user viewing the page is a bot (by IP, or special headers, there was sth on bots here), and give it access (href link) to the pages you want to index.

I don't know of any other way of making such pages indexable.

Hope this answers your thread.

And welcome to WebmasterWorld!

Regards
Michal

aff developer

5+ Year Member



 
Msg#: 3140406 posted 1:49 am on Nov 5, 2006 (gmt 0)

thanks for that. Is it possible to provide bots such as googlebot with a cookie for your pages so they can login and then index the pages if they are current restricted by a required login?

Also would this be considered cloaking as your not really giving the end user the actual page that would be indexed (assuming they don't have a cookie stored)?

Any help and info would be great.

mcibor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3140406 posted 10:30 pm on Nov 7, 2006 (gmt 0)

Everything about googlebot:
[google.com...]

There is also some info on
[en.wikipedia.org...]

And
[en.wikipedia.org...]

As you see there is no cookie included, as the crawlers are not run through browsers.
So if you spot a crawler - by IP and / or agent's name you can override you security and grant him access to some pages you want to index.

Hope this helps you
Michal

aff developer

5+ Year Member



 
Msg#: 3140406 posted 11:03 pm on Nov 7, 2006 (gmt 0)

Wouldn't this be classed as cloaking though?
It would be pretty easy with my script to set it via useragent to let Google crawl my members forum, and then set the nocache option - but isn't that against there tos (cloaking?)

Thanks

mcibor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3140406 posted 9:26 am on Nov 11, 2006 (gmt 0)

It may be referred to as cloaking and probably will

[google.com...]

Why do you want grant google access and users not?

aff developer

5+ Year Member



 
Msg#: 3140406 posted 10:01 pm on Nov 11, 2006 (gmt 0)

I do want to allow both however my site is a paid subs site (same as WebmasterWorld) so obviously these pages will be blocked for users that dont have the correct cookie.

It is more like IP cloacking (which Google say is legal) as we are allowing users that have a cookie (at certain) IPs to view the pages and others will have to join to view.

Very hard to get the answer out of Google. Hopefulyl GoogleGuy could lend his 2 cents?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved