Forum Moderators: open

Message Too Old, No Replies

Is returning a blank page to known search crawlers acceptable?

Filtering search crawler clicks to affiliate links

         

JS_Harris

12:13 pm on Aug 21, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I'm building an affiliate site/tool in php that has several value added features besides just an api call and data parsing. One functional feature that i've built breaks the product link provided by the xml datafeed into segments and passes it on to a second php page which processes any clicks on those links. This allows me to point all links to this single page for processing which means I can make my on-site links appear much cleaner than traditional long affiliate links.

My question is: Most reputable search crawlers will obey either the nofollow link tag, nofollow page meta tag or robots.txt exclusion which are all in use on these links and the link processing page. Would it then be an acceptable practice to filter other known crawlers by user agent and send them to a blank page instead of through to the affiliate site?

My concern is getting into trouble with cloaking or other related rules which could lead to being de-indexed. Google for example is strict on not showing crawlers something different from what is shown to visitors, i'm not sure if that applies to what happens on the other end of outgoing clicks.

In case it matters the site does not use a database, crawler filtering would be incorporated into the link processing/redirect page.

wilderness

1:39 pm on Aug 21, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



It really should NOT be an issues.

What would be the difference between your method and either denying them access or sending them to a null?
What would be the difference between your blacklisting method and whitelisting?

Your not redirecting the major bots, however these days, many of the major bots are using cloaked UA's. As a result, you need to be sure that your not sending one of the major bots to your blank page.

Don

JS_Harris

12:26 am on Aug 23, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks Don, good points.

I'm not overly concerned with sending major bots using cloaked UA's to the blank page because if they get there they ignored the link nofollow tag and loaded a noindex page that was also disallowed by robots.txt. Shame on them.

I am concerned with the potential for my filter to block a non cloaked UA while allowing a cloaked UA to pass from the same search engine. If they compare notes then I'm concerned with an automated "you're showing different pages to crawlers" possibility of getting penalized. I'm confident I could explain what I did and why but I'm not so sure the search engine could explain ignoring all my "don't go there" measures.

I don't anticipate any problems so i'm going ahead with it, keeping an updated list of UA's should be enough.