How to Prevent Google from Following Certain Links

Want to Avoid Potential Bad Neighborhoods

[jimh009]

Hello,

I have a content site that provides very targeted information. That same content is beginning to lead businesses, which provide services directly related to the content, to want to advertise their services on my site (which was one of the goals of my site). These business listings include a description of their business as well as a link to their web site. This is a paid listing (or will be once the free period ends). Thus, this is not a link exchange type thing done for PR purposes.

Here's my dilemma. Some of these sites, more likely due to bad design then intentional design schemes, are PR0's. So, to avoid any "bad neighborhood" penalties on my own site, I need a way to prevent Google from following these particular links. However, I want Google to be able to follow every other link on my page, so a robots.txt exclusion won't work.

I'm far from an expert coder, so have no idea how to set up something like this.

Any ideas, hopefully explained so dummies can understand, would be immensely helpful.

Thanks.

Jim

[Marcia]

Hi, jimh009. I suppose it's *possible* but it's not too likely that they're PR0 for poor design. Are there any links to them at all? If there are, they should have some PR.

I'm assuming you're selling them advertising space, and not PR. Good - then you need some tracking of how much traffic you're sending them. For one thing, you'll need those statistics for future space sales and you could be under-selling price wise. You need to know.

They may, in fact, be looking for PR but that's inconsequential; it has to be done for informational purposes. Head over to alltheweb.com and check their links, as well as where their URL or name is mentioned on web pages. You might just find that they do have inbound links in spite of the PR0. Maybe not, but it's better if you do know what the situation is.

It's not in the HTML code, and you can find Perl scripts out there that are simple to use for this purpose. Check a few categories at hotscripts.com including redirection and you'll probably find just what you need.

[jimh009]

Hi Marcia,

Yes, I'm selling them advertising space - although it's more of a "listing" (100 words, plus image, plus web link) then a simple type banner ad. The state I live in has lots of businesses that cater to the content of my site, hence the interest in it.

Some of these sites are actually new, which is one reason they have a pr0. But one site has two "entry pages" into it, which is what got me concerned. One entry page has a PR0 and the other a PR5. I linked to the PR5, but it struck me that with two entry pages, both could turn to PR0 in a heart-beat. Hence, the reason to be safe and prevent Google from looking at the link.

Your idea of tracking how often the link is checked is a good one - didn't even think of that. I'm selling this ad space on a yearly basis for a flat fee, so its not PPC. Still, will be nice to know how effective the ads are.

Thanks for the tip about hotscripts.com

Do you think just using javascript for the link would work well - since Google doesn't follow Javascript links? Or do you have any recommendations of scripts out of the many that are listed at hotscripts.com? I didn't see any that also would track how many times a link was clicked.

Thanks and sorry about the stupid questions!

Jim

[dingman]

I don't think I'd pay as much for a Javascript link as I would for a redirect link, simply because there are browsers in which they won't appear, either because the surfer has deliberately disabled Javascript or because they are using a browser that doesn't support it. (I know text-mode browsers are an insignificant niche, but think about cell-phone browsers. I've got a pretty advanced one with a 320x240 screen and it'll do images, but no javascript.) On the other hand, any browser at all can follow 'http://your-server/redirect/redir.cgi?client=#####' and a 302 redirect. Add a prohibition on the /redirect folder in robots.txt, and you should be golden.

[jimh009]

Thanks dingman,

Any suggestions on recommended CGI scripts to use - that will run in a Windows environment? There was a ton of them at hotscripts.com, and I really couldn't figure out which ones would work well and one's that won't - or how to get them to work, as I've never used CGI before.

CGI and Perl are not my strong suits.

Thanks.

Jim

[dingman]

That I can't offer so much advice on. For something that simple I'm far more likely to write it myself than to use something someone else wrote. A vague outline in PHP might look something like:

<?php 
ob_start(); //Probably won't matter, but makes sure header() will work. 
/* log whatever you want to keep track of here */ 
if (preg_match('/(Googlebot¦OtherBotWeCareAbout)/', $_SERVER['HTTP_USER_AGENT'])) 
{ 
 header("HTTP/1.1 404 Not Found"); 
} 
else 
{ 
 header("HTTP/1.1 303 See Other"); 
 header("Location: ${_GET['target']}"); 
} 
ob_end_flush(); 
?> 
<html> 
<head> 
<title>Whatever</title> 
<meta name="robots" content="nofollow" /> 
<meta name="robots" content="noindex" /> 
<meta name="robots" content="noarchive" /> 
</head> 
<body> 
<!-- standards say there should be some HTML here, but no  
user agent ever shows it anyway. --> 
Redirecting to <a href="<?php echo $_GET['target'];?>">requested site</a>... 
</body> 
</html> 

If that file were redirect.php, a redirected URL would look like http ://your-server/path/to/redirect.php?target=<url-encoded-client-url>

This code is 100% untested, and I'm tipsy, but it probably comes close to working. It doesn't actually log anything, though. What you want to log would be your choice.

[jimh009]

Thanks dingman.

I'll give it a try and see what happens.

Jim

[dingman]

Just tested it. It's a bit brittle, but it does work. Ceveat emptor, though. If you don't url-encode the argument to target, you'll probably create an infinite loop. The one browser I tried that in got out of it very quickly, but even so the effect was not the desired one.

Overall, not too shabby for code I wrote while tipsy and without looking at a manual. Maybe I'll polish it someday, when I have propper motivation.