A file on a web site in the root directory of a website that is used to control which spiders have access to which pages within a website. When a spider or robot connects to a website, it checks for the presence of a robot.txt. Only spiders that adhere to the Robots Exclusion Standard will obey a robots.txt command file There are several specific fields in a robots.txt such as User-agent specifies which User Agents are allowed to access the site and "Allow/Disallow" specifies which directories a spider may access.
For full specifications see Altavista.
