Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
For instance if you didn't want to have your visitor stats crawled by the search engines, perhaps you would forbid the directory stats/ or where ever your stats page lives.
try [robotstxt.org...] or maybe
This is assuming they follow robots.txt protocols. Don't rely on robots.txt file alone to protect sensitive information of temporary folders/files.