Another question I have... I have successfully created a robots.txt file, great - and it works (have tested it with the bot sim), etc ... but i want to know how I can check the log files on my domain - is there some sofware I can download? - at the moment I am unsure whether my site is being spidered?, thx
This depends on your hosting service. If your site is small, you can usually just use your browser to view the raw log files in a secure subdirectory of your site, or download them using ftp. No special software is required. The internet explorer browser supports ftp access as well as normal browsing.
I'd suggest looking at your host's help pages, or looking in your site's "control panel" if it is set up with one.