tedster - 4:41 am on Jun 13, 2012 (gmt 0)

Looks like a misunderstanding. deadsea said that you SHOULD have a robots.txt file that says:

User-agent: * - but he forgot a line. g1smd then gave the correct syntax in his follow-up post. That syntax allows everything to be crawled. Then, if you have other needs, you can develop from there.