Welcome to WebmasterWorld Guest from 54.196.243.192

Forum Moderators: open

Message Too Old, No Replies

Getting subscription pages indexed

     
8:26 am on Mar 15, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:May 8, 2002
posts:440
votes: 0


I know this is a bit of a contradiction but has anyone found a way round this: getting google to index pages that are for members only; or, as I have on one site, pages that are cookied to allow only one page view per day.

Anyone tried, for example, looking out for googlebot in page code and serving it the full page, instead of redirecting to the login or whatever?

Many thanks

4:26 am on Mar 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 7, 2003
posts:138
votes: 0


Webmasterworld seems to, and the way you want to do it is probably very similar to how it does it.

Server the full page to the google bot, but tell it not to make the cached page available. Otherwise people can just look at the google cache.

4:43 am on Mar 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 22, 2002
posts:65
votes: 0


You may want to have it look for "googlebot.com" instead of just the user agent string because anyone can spoof that. And be sure it block the snippets like ddent said.
4:58 am on Mar 16, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 6, 2002
posts:4768
votes: 0


I really have a problem with this. Google is a SEARCH ENGINE, not a mechanism for you to advertise your members-only content.

What part of WebmasterWorld is spidered but is unavailable to non-members? I can seem to access it pretty well without signing in.

5:03 am on Mar 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 7, 2003
posts:138
votes: 0


The forums.

Maybe its just due to my ISP (though its a pretty large ISP, and they would have other means of tracking abusers i.e. static hostnames), but for months many of the searches I did would turn up hits on webmasterworld, but when I clicked on them I was told I had to log in to read the message. Eventually I decided I may as well create an account. :)

5:19 am on Mar 16, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member bigdave is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 19, 2002
posts:3454
votes: 0


Never had a problem with getting to WW forums before I became a member.

I'm with jomaxx. I would find it incredibly annoying if I came across a site doing this. You are wasting my time and reducing the quality of the search results.

5:26 am on Mar 16, 2003 (gmt 0)

Administrator

WebmasterWorld Administrator rogerd is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 2, 2000
posts:9685
votes: 0


Although I've never observed this, I think Brett has at times required users from certain domains to log in to prevent abuse issues.
6:10 am on Mar 16, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 7, 2003
posts:138
votes: 0


That would coincide with the message I was getting.

FWIW, hostnames (though not the associated IP addresses, however the reverse of the IP is updated constantly) ending in .bc.hsia.telus.net or .ab.hsia.telus.net will be constant and pretty much assured to be one customer. Most customers will have two hostnames, though some will have five.