sftriman - 11:39 pm on Jan 24, 2012 (gmt 0)
I have lots of ebay on my site, and some affl relationships with AvantLink, ShareASale, CJ, and others. My ebay links go through redir.php, while all other affl links go through refer.php. For each, I log data from the CGI variables, then grab (or format) the URL, and redirect to that page. In each case, ebay and affl, that next link is itself a redirect, of course.
Well, I put redir.php and refer.php in my robots.txt file so that googlebot would ideally not crawl them. Unfortunately, I made a typo in the file, so googlebot did crawl them. They actually are really slow since CJ in particular redirects really slowly. So in Google Webmaster, my site speed was slow b/c of those links.
I fixed my robots.txt file, and figured after a few months, my site speed score would not include redir.php and refer.php as example. Nope. They are still there.
In light of everything I'm reading here, I'm thinking about making a change. I'll rename both the scripts, so that I'm essentially starting over with the redirects on my site. I'll make sure that robots.txt is correct. Of course, other bots ignore that file and crawl anyhow. But to specifically make sure googlebot doesn't crawl those links, I was thinking to block it in my .htaccess file.
One might say, I should not block google from seeing what users see. But the thing is, google should not crawl them, via my robots.txt setting, and I also put rel="nofollow" on every single link as well. By blocking googlebot (and other bots I identify) from crawling them, could that safeguard my site against being seen as an ebay/affl site?
I don't think this is black hat, by the way, not at all. If I have 20,000 pages, google can crawl and index all 20,000 pages. I just don't want my redir links crawled or counted. Since they do not appear directly as ebay or affl links to google, that might/should help out my site. I'd think, at least.
In the meantime, yes, I need to continue to add content and value to the user beyond just having links to products. I get that. If I continue to be pandalized b/c I don't add content, I think that's fair enough by their rules. I just don't want to get additionally penalized just because of my redir links.
Any thoughts on this?