Their is info at w3.org [w3.org...] scroll down the page to "notes on helping search engines index your web site."
I also found this robots.txt checker: [tardis.ed.ac.uk...] This page also offers "botwatch". The key to the robots.txt file is to make sure the syntax is written exactly. Otherwise the robot can't read it because it's a text input file.
Does this help at all? I'm just learning and gathering resources at this time, as you probably know. This is so cool. :)
no problem BH. check this flash site out, m-axis.com. make sure you have your sound on!
wish someone would answer your cgi question soon. oh, i think i know. the robot doesn't get the cgi program of course, they get the html page you decided to serve them through the cgi script's logic. you know, the client sends out a message to the server and the server then sends them a message and a html file. so every message sent to the server is filtered through the cgi program first, so i highly doubt their is a way for them to tell, unless you tell them. hehe. is this dull?
no, that isn't mine! i shutter to think how long it would take me, but i think it's a good example. something to strive towards. one should probably 'begin' with a graphic art's degree as the graphics are outstanding, to say the least.
Thanks for the sound link! i have been thinking about sound but i'm still on 'day 2' since i became side tracked due to the search engine problem. i don't want to write a program that has to consider every single agent on the planet. i think i will just have a default solution after the most obvious has been considered, or something??