Forum Moderators: open
My question is: Most reputable search crawlers will obey either the nofollow link tag, nofollow page meta tag or robots.txt exclusion which are all in use on these links and the link processing page. Would it then be an acceptable practice to filter other known crawlers by user agent and send them to a blank page instead of through to the affiliate site?
My concern is getting into trouble with cloaking or other related rules which could lead to being de-indexed. Google for example is strict on not showing crawlers something different from what is shown to visitors, i'm not sure if that applies to what happens on the other end of outgoing clicks.
In case it matters the site does not use a database, crawler filtering would be incorporated into the link processing/redirect page.
What would be the difference between your method and either denying them access or sending them to a null?
What would be the difference between your blacklisting method and whitelisting?
Your not redirecting the major bots, however these days, many of the major bots are using cloaked UA's. As a result, you need to be sure that your not sending one of the major bots to your blank page.
Don
I'm not overly concerned with sending major bots using cloaked UA's to the blank page because if they get there they ignored the link nofollow tag and loaded a noindex page that was also disallowed by robots.txt. Shame on them.
I am concerned with the potential for my filter to block a non cloaked UA while allowing a cloaked UA to pass from the same search engine. If they compare notes then I'm concerned with an automated "you're showing different pages to crawlers" possibility of getting penalized. I'm confident I could explain what I did and why but I'm not so sure the search engine could explain ignoring all my "don't go there" measures.
I don't anticipate any problems so i'm going ahead with it, keeping an updated list of UA's should be enough.