Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: goodroi
For instance if you didn't want to have your visitor stats crawled by the search engines, perhaps you would forbid the directory stats/ or where ever your stats page lives.
try [robotstxt.org...] or maybe
This is assuming they follow robots.txt protocols. Don't rely on robots.txt file alone to protect sensitive information of temporary folders/files.