cgi-bin and robots.txt

Should you deny access?

[billg51]

If you are sending outgoing links out through a cgi link, or having other link to your site with a cgi link, should you not block the cgi-bin through robots.txt?

I just don't know the ramifications if you don't deny access.

Thanks.

[diddlydazz]

hi billg51,

if you don't want googlebot to follow the links then ban it via robots.txt to be on the safe side.

Dazz

[ncw164x]

Why just goolebot, this applies to all spiders and robots (if they abide by the robots code) Your cgi-bin can contain scripts for all kind of actions for your web site, from a simple email form to complex database scripts. That is why you would normally ban access

[ciml]

I don't see how blocking robot access to your outbound links helps.

If you link to someone who links back, then an improvement in their PageRank may help them get more people to follow your link. Also, some of your PageRank should be passed back to you if they link back.

If you link to a robots.txt forbidden URL, or a dead link (404), then the PageRank is lost.

[ncw164x]

<snip>I don't see how blocking robot access to your outbound links helps<snip>
ciml
Are you meaning as an example www.example.com/cgi-bin/nameofscript.cgi?id=123456 which say goes to www.webmasterworld.com.

On your html page you would have the text "Webmasterworld" as a hyperlink via the above example which goes through your cgi-bin

If a site is made up of links via cgi scripts it is the web pages which are indexed (with the individual links on the page) and it is the either the name used for the hyperlink or the keywords which show up on the search results page at the search engines and not "http://www.example.com/cgi-bin/nameofscript.cgi?id=123456"

The search engine spiders index your html page which has the word Webmasterworld.
Even though you banned access to your cgi-bin via the robots.txt file, the page "hopefully" will be found by someone searching for the word "Webmasterworld".

<snip>If you link to someone who links back, then an improvement in their PageRank may help them get more people to follow your link<ship>
They would be linking to your html page which shows their link (home/index page) and likewise you would be linking to their html page showing your link (your home/index page)

When googlebot visits either your site or the site you link to it will pick up both of the pages and do a Page Rank

Blocking robot access to your outbound links via your cgi-bin in the robots.txt file does not seem to make any difference because it's the page the link is on which gets the PR, but maybe someone can be prove me wrong?

[edited by: ciml at 5:36 pm (utc) on Mar. 6, 2003]
[edit reason] url fixed [/edit]

[ciml]

From a WWW point of view, "goes through your cgi-bin" would indicate an HTTP redirect. If it's a 301 redirect then it probably can pass PageRank.

I agree that the words on your page can be indexed, but you might as well donate some PageRank to the people you link to. If the script runs at a robots excluded URL, then that PageRank would be lost anyway.