Credit for links: does Goog barf on this?

Need advice/direction to thread on link popularity

[MrSniffles]

Our company has referring links that don't get credit with Google. We use cgi-bin to check for a cookie, set a cookie if they don't have one, and then check the cookie again. Google gets a 404 because of this. How can we structure links so that we get credit for referring links? Would setting the cookie later be a good solution, or would dumping cgi-bin altogether be better? I appreciate any input or direction to a thread that has answers.

They look like:
href=http://www.widget.com/cgi-bin/CDcart?REF=myrefcode&DEST=/cgi-bin/MySQLdb%3fMYSQL_VIEW%3d/cart/cat/view_my_page.txt

[whoisgregg]

Why don't you first check if it's a google user agent and display the page, then if it's not google set a cookie for a user? You can do this by checking for a google user agent or by IP (if you end up having users spoofing user agent to circumvent your controls).

Many forums do this to allow Google to spider content that's normally restricted in some way to humans.

[MrSniffles]

Thanks for your reply - that sounds like a good idea. How would I implement something like that/where could I go to read up on that?

[MrSniffles]

Do you mean like setting it so that visitors using browsers like googlebot's browser (Googlebot/2.1)
or MSN's msnbot/0.11 wouldn't have a cookie set on it?

[whoisgregg]

Precisely. If they are reporting a user-agent accurately, you shouldn't need a tracking cookie to know what they do on your site.

I don't have sample code for you, I don't do perl. Since you are using a perl CGI already, you should post this request in the perl forum [webmasterworld.com]. There are some very helpful folks there. :)

You may also try a site search for "perl user agent detection" with your favorite search engine. (In Google, for example, use "site:webmasterworld.com perl user agent detection")

[conor]

Try using ´spider food´