Msg#: 3140406 posted 12:11 am on Oct 31, 2006 (gmt 0)
I had a quick question and couldn’t find it anywhere on the forum and was wondering if anyone would know please?
I am just wondering how you get all your forum pages indexed (like in webmasterworld.com) in the search engines without having the cache pages viewable as well? Also how the googlebot gets into the forum without having a cookie set? I would be keen to do this to my small forum that I keep as a paid only forum and I want to respect the members by not making all the information within it public, but still being able to generate traffic from search engines. Any help would be greatly appreciated!
Msg#: 3140406 posted 9:16 pm on Oct 31, 2006 (gmt 0)
As I remember, the spiders only index non restricted pages. However if you want to give them access to your more privat stuff, then you would have to check if the user viewing the page is a bot (by IP, or special headers, there was sth on bots here), and give it access (href link) to the pages you want to index.
I don't know of any other way of making such pages indexable.
As you see there is no cookie included, as the crawlers are not run through browsers. So if you spot a crawler - by IP and / or agent's name you can override you security and grant him access to some pages you want to index.
Msg#: 3140406 posted 11:03 pm on Nov 7, 2006 (gmt 0)
Wouldn't this be classed as cloaking though? It would be pretty easy with my script to set it via useragent to let Google crawl my members forum, and then set the nocache option - but isn't that against there tos (cloaking?)