pageoneresults - 11:41 am on Apr 26, 2011 (gmt 0)
One more thing about robots.txt files. Googlebot will discover URIs via this method. Yes, they will get in there and start crawling URIs that it can find via robots.txt. One look in your GWT and you can see what they are crawling. In some instances, you may discover technical issues you were not aware of. In others, you'll find the bot has crawled what you "thought" you told them not to crawl. Just keep in mind, we're discussing a crawl and not an indexing, they are different.
Do this, take an item from your robots.txt file that is capable of generating thousands of pages. Now, do a site: search for that specific path. What did you find? One URI only entry with a link to show omitted results? Okay, how many omitted results were there? Now tell me, why would you want those thousands of URI only entries available for someone to scrape? It's a road map for everything you don't want indexed.