I currently have a couple of CGI scripts that create relatively static pages from a database. They are performed dynamically because they are shop products, there are over 3000 products, so static files are not really feasible.
Google cannot spider the cgi-bin as my host has disabled that by default (nor can people visit from google's result - I can see google has attempted to spider as it lists the cgi files when I search on my domain name).
Is it possible to allow public access to the CGI files, with the referrer checking off? Can it be done in .htaccess?
I have seen:
<Directory "/web/cgi-bin"> Options ExecCGI Order allow,deny Allow from all </Directory>
Would this be correct? Or is there a better way, where I can specify individual files rather than a directory?
The easiest and safest way to do this might be to simply put those scripts into a different directory. The placment of CGI scripts into cgi-bin is more of a convention than a requirement... unless your hosting company requires it. I'd suggest talking to your hosting company if running scripts in other directories doesn't work.