If you dont have a robot.txt the search engines will just index your site, Why have the robots.txt file
1:25 pm on Feb 21, 2005 (gmt 0)
You may not want certain/all search engines to see certain/all pages on your website.
8:43 pm on Feb 25, 2005 (gmt 0)
Robots.txt will help keep SEs out of directories you don't want indexed. For example, someone who sells his images might not want google's imagebot to pull all of his images out of a directory and list them where they can be freely downloaded.
8:59 pm on Feb 25, 2005 (gmt 0)
And having any robots.txt file --even a blank one-- will prevent a lot of 404-Not Found errors in your logs, thus making it easier to identify real problems.