Affiliate links, a valid referrer and googlebot

[Sgt_Kickaxe]

I'm making an improvement to my site in how I handle affiliate links and I just wanted to get some feedback on potential problems you would worry about with these changes re:googlebot.

Old way:
- visitors clicking on affiliate links must pass a referrer check to make sure they are ON our site. This is done because some affiliate sites penalize and ban for sending traffic from a site we don't own. Googlebot obviously fails the referrer check and like anyone failing that check it sees a blank page on our domain instead.

noindex meta tag on blank page

Possible new way #1:

- visitors still need to pass a referrer check but when the check fails, as it will for Googlebot, the blank page is replaced with an explanation of why they are seeing that page (failed referrer check, ip in banned range etc) along with a contact us form. A canonical tag would point to just one version of the blank page since there would be hundreds known to google, like there was with blank pages.

noindex meta tag applied to this page as well

Possible new way #2:

- same as method #1 with the exception that instead of showing hundreds of these pages for the various redirect urls all the redirect pages would 301 failed referrer checks to ONE copy of the page. googlebot would thus be redirected as well.

noindex meta tag applied to this page as well

Being that both new methods maintain the noindex tag this might be a moot point but the benefits to #1 are not redirecting googlebot to a different page than visitors who pass the referrer check (googlebot would also pass if it provides a valid referrer).

The downside to method one is that googlebot still visits hundreds of these pages instead of just one. Method two would condense all of these pages into one but then the worry becomes a penalty for redirecting googlebot to different content than regular visitors (even though it's googlebots fault for not providing a valid referrer).

Unfortunately the referrer check is not optional. The site does not however perform a user agent check, googlebot and visitors alike can pass regardless of user agent.

Thoughts?

[phranque]

don't you want googlebot to see a rel=nofollow on those links?

[Sgt_Kickaxe]

On content pages, yes, but I'm concerned with the redirect pages and the blank pages Googlebot receives. GWT reports all the blank pages as not chosen for example and they waste considerable crawl time looking up urls that are noindex'd and will never have content. I tried serving up 404 errors instead of blank pages but that was a disaster, it made googlebot think all my links were broken.

- Is it ok to serve googlebot a blank page instead of redirecting it to the affiliate destination? (standard behavior on some popular affiliate software packages).

- Is it ok to use rel-canonical on the blank pages, pointing them all to one blank page, so that they are all at least concentrated into *hopefully* one page?

- or is it ok to redirect googlebot to one blank page instead of serving a blank page for every jump url?

I like the last option best but does it run the risk of running afoul because I send Googlebot(and any visitor who fails the referrer check and isn't on my site) to one url and visitors to another?

[xcoder]

Googlebot obviously fails the referrer check and like anyone failing that check it sees a blank page on our domain instead.

Not necessarily. Googlebot often pass a referrer and render the page just like any other web browser would (it's a chrome hybrid). You need to run some kind of a "human" test to be absolutely sure.

[Sgt_Kickaxe]

No human test needed since I only want to be sure the referring url belongs to my site. e.g. They clicked the link on my site and not on another domain.

Since Googlebot does both apparently it sometimes gets a blank page and sometimes passes to the destination. Is this a dangerous thing? If it's not I'll go with option #2 and stop showing blank pages, I'll redirect fails to a single blank url instead.

I'm not sure Google spam doesn't make a distinction between being shown a blank page and being redirected to one.

[phranque]

Old way: - visitors clicking on affiliate links must pass a referrer check to make sure they are ON our site. This is done because some affiliate sites penalize and ban for sending traffic from a site we don't own. Googlebot obviously fails the referrer check and like anyone failing that check it sees a blank page on our domain instead.

noindex meta tag on blank page

noindex solves the "duplicate blank content" problem and there are no links to follow so it blocks googlebot from seeing the affiliate link.
as long as google doesn't see this as cloaking it should be ok.
you might even add a X-Robots-Tag: noindex header just to drive the point home.

i would suggest the proper response here would actually be a 403 Forbidden status code and it would probably achieve the same affect as the referrer cloaking and meta robots noindex.

Possible new way #1: - visitors still need to pass a referrer check but when the check fails, as it will for Googlebot, the blank page is replaced with an explanation of why they are seeing that page (failed referrer check, ip in banned range etc) along with a contact us form. A canonical tag would point to just one version of the blank page since there would be hundreds known to google, like there was with blank pages.

noindex meta tag applied to this page as well

never explain your rejections - it might only encourage a more targeted reaction to failed vulnerability probes.

this is really like a "soft 403" - it's essentially a Forbidden response hiding behind a 200 OK status code.

the meta robots noindex makes the link rel canonical moot.

Possible new way #2: - same as method #1 with the exception that instead of showing hundreds of these pages for the various redirect urls all the redirect pages would 301 failed referrer checks to ONE copy of the page. googlebot would thus be redirected as well.

noindex meta tag applied to this page as well

redirecting a large number of urls to a single url would be a signal of low quality.
perhaps the noindex would bail you out from that.


i think you would be better off with, in order:
- rel="nofollow" attribute on those links and a 403 response for no Referer header
or:
- excluding the urls from crawling in robots.txt