petehall - 11:34 pm on Dec 19, 2006 (gmt 0)

Depending on how your site is designed, you need a robots.txt file. For example, most CMS have areas that should be off limits to robots. If you're running a site that uses mod_rewrite, you also need robots.txt to prevent duplicate content.

If a robot can gain access to a CMS area then you have severe security issues to resolve :)