Page is a not externally linkable
-- New To Web Development
---- Spider Link Following
Dreamquick - 2:20 pm on Jan 12, 2003 (gmt 0)Has /cgi-bin/ been excluded via robots.txt? Potentially if this isn't your site then someone could create a link out script and then block spiders from the cgi-bin directory either for a genuine reason (ie maybe it affects their tracking stats) or in an attempt to weasel their way our of giving a "real" link.
Two things immediately spring to mind;
Are the URLs overly long? Lots of data in a querystring will often deter a search engine from crawling a certain URL - mostly due to a fear of hitting infinitely dynamic pages. That said though there are example of people who say that google did crawl similar links so you might be safe...
Thread source:: http://www.webmasterworld.com/new_web_development/253.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com