Forum Moderators: open
We have a business website that offered free access to visitors for the last 3 years. It's on the top-5 results of Google for a very competitive business word and now we have developed a membership system within PHP and MySQL that will restrict access to members for specific pages OR provide free access to visitors but will also catch whether they are members or not ("Welcome Guest" or Welcome YourName" will be displayed on free-accessed pages).
For this reason we will be using a PHP script on the top of each page (free and restricted) that will check whether a visitor is already registered or not, based on a cookie.
I would appreciate if you could share your experiences on how Googlebot will probably behave on this major change of content structure and how we could be sure that all the pages (can we confirm it for restricted pages also?) will be crawled, so we will not loose our high ranking.
Thanks for all your help!
Kaled.
Our website has 15,000+ pages of unique content, so we want to be sure that members-only content will not result in not-crawling by Google.
Thanks
Anyway, I am still trying to find some friends to share their experiences on this.
Thanks
Otherwise, nothing much should change if you are only adding a line with the username.
Is there any way to leave specially Googlebot to crawl the pages that require login?
If you have written a membership system in php, you should be able to figure out the details of this without difficulty.
To prevent member pages being cached by google, you will have to ensure that each is delivered with the robots meta NOARCHIVE (not nocache as I suggested above).
Kaled.
In other words, if I follow a link from google to any of your pages, would I get the page I am looking for, or would I get a login page?
If it is the latter (i.e. login REQUIRED) then you can expect that you will get yourself in trouble by allowing googlebot, but not a surfer to access the page.
You should always depend on your free content to attract users through search engines. Users don't like the first page they get on a site to be a login/register page. And if users don't like it, then google won't like sending users there. I have never seen a site that cloaks to allow googlebot into member only pages that lasts more than a few days.
If you are just calling a user without a cookie "Guest" but still allowing access to the page, then you should be just fine.
I have never seen a site that cloaks to allow googlebot into member only pages that lasts more than a few days.
It's frequently done with good results, but it is definately black hat and can land you in trouble. My advice is: don't do it - make enough of the site free that you keep your keyword positions.
Thank you all for sharing your thoughts on this. Let me try to make it more clear:
Our site has more than 15,000+ pages of unique-real content and our new membership script (I am not a/the developer) will divide pages into 2 categories:
1. Pages that are free to read (no login required) but there will be a cookie-check in order to welcome our registered members (i.e. "Welcome BigDave" or "Welcome Guest" if no cookie available).
2. Pages that will require a login in order to be viewed. This category will certainly include the minority of our pages.
So I wonder the following:
- Will the 1st category have any problem with Google being indexed?
- Is there any way (legal and without ANY risk to get penalty!) to allow Google to crawl successfully the 2nd category pages?
Please note that joining our site is free (maybe this is helpful) and furthermore our site - as many others! - tries to be very credible; for this reason we would never like to risk any kind of penalty by Google or any other search engine.
Does Google considers such a googlebot manipulation as spam or reason-for-penalty? I mean allowing googlebot to access these pages (what if letting it archive it also? does it make any difference).
THANKS again for your help. I am the founder of the site but don't have much experience on such Googlebot issues.
Best Regards from Athens, Greece
Aris
PS> If you can share some similar thoughts about Adsense impact of the above scenario it would be very useful too.
- Will the 1st category have any problem with Google being indexed?
No, unless you are doing something stupid with the code. Just let them be browsed as guest.
- Is there any way (legal and without ANY risk to get penalty!) to allow Google to crawl successfully the 2nd category pages?
I sure hope not.
Please note that joining our site is free (maybe this is helpful)
It doesn't matter in the least.
Think of this from a searcher's perspective. They see a somewhat interesting listing in google. They click on the link. They get a page that is totally unrelated to the page they were looking for asking them for personal information.
Your page is not giving them what they are looking for. It is not giving them what you showed to googlebot.
The surfer has not had a chance to find out if they would like to become a member of your site before you ask them to register.
What is the likely response to your site? The back button in probably 98% of the cases.
Does this sound like the sort of thing that google wwould want to consider as a top 10 result?
Remember, you have to sell your site to the user if you want them to put in the effort, and what you are trying to do is wasting everyone's time and effort.
and furthermore our site - as many others! - tries to be very credible; for this reason we would never like to risk any kind of penalty by Google or any other search engine.
Well, what you are suggesting is not credible.
Does Google considers such a googlebot manipulation as spam or reason-for-penalty?
Yes. You are serving something different to the bot than you are to the user.
Put up some intro pages to the full articles, and let the bot crawl those.
Your last note is really what I was ready to do, but I was still interested in understanding the problem with google and member-only pages.
You were very clear and it seems I am "forced" to fully agree with your perspective. :)
I do appreciate your valuable feedback.
