Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: mack
Can search engine robots follow links like this or does the '?' scare them off?
Actually I'm hoping they can follow, I'm under the impression Google likes sites to link out.
It doesn't necc have to "click" the link to "follow" the link (unless you encode the link).
David was talking about using the cgi-bin directory for links, specifically using a cgi script to link out to others. Now although the link-thru looks obvious enough it is realistically the same as using an encoded value.
Now unless I'm missing something...
Although a spider may think that "www.example.com/cgi-bin/linkout?http;//test.example.com" would link to "test.example.com", it can *never* be sure without requesting that script because it's dealing with a dynamic server-side script without any idea of what the code behind that script actually does.
Even if cgi-bin is blocked by robots.txt, Google will read the following link to bar.com
I agree it will read/note the link and potentially include it in the index but are you saying it will access the redirecting script (which it thinks points to "bar.com") violating robots.txt in the process?