robots.txt is a config file generally located in the root directory of web site. There are certain parameters which specify how a search engine robot (or any other robot) will crawl the site. You can specify some directories or file not to be crawled by. You can also allow/disallow a specific robot (identified by user agent). Search by "robots.txt" in this site, you will find lots of important information.
