homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Getting subscription pages indexed

 8:26 am on Mar 15, 2003 (gmt 0)

I know this is a bit of a contradiction but has anyone found a way round this: getting google to index pages that are for members only; or, as I have on one site, pages that are cookied to allow only one page view per day.

Anyone tried, for example, looking out for googlebot in page code and serving it the full page, instead of redirecting to the login or whatever?

Many thanks



 4:26 am on Mar 16, 2003 (gmt 0)

Webmasterworld seems to, and the way you want to do it is probably very similar to how it does it.

Server the full page to the google bot, but tell it not to make the cached page available. Otherwise people can just look at the google cache.


 4:43 am on Mar 16, 2003 (gmt 0)

You may want to have it look for "googlebot.com" instead of just the user agent string because anyone can spoof that. And be sure it block the snippets like ddent said.


 4:58 am on Mar 16, 2003 (gmt 0)

I really have a problem with this. Google is a SEARCH ENGINE, not a mechanism for you to advertise your members-only content.

What part of WebmasterWorld is spidered but is unavailable to non-members? I can seem to access it pretty well without signing in.


 5:03 am on Mar 16, 2003 (gmt 0)

The forums.

Maybe its just due to my ISP (though its a pretty large ISP, and they would have other means of tracking abusers i.e. static hostnames), but for months many of the searches I did would turn up hits on webmasterworld, but when I clicked on them I was told I had to log in to read the message. Eventually I decided I may as well create an account. :)


 5:19 am on Mar 16, 2003 (gmt 0)

Never had a problem with getting to WW forums before I became a member.

I'm with jomaxx. I would find it incredibly annoying if I came across a site doing this. You are wasting my time and reducing the quality of the search results.


 5:26 am on Mar 16, 2003 (gmt 0)

Although I've never observed this, I think Brett has at times required users from certain domains to log in to prevent abuse issues.


 6:10 am on Mar 16, 2003 (gmt 0)

That would coincide with the message I was getting.

FWIW, hostnames (though not the associated IP addresses, however the reverse of the IP is updated constantly) ending in .bc.hsia.telus.net or .ab.hsia.telus.net will be constant and pretty much assured to be one customer. Most customers will have two hostnames, though some will have five.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved