g1smd - 8:57 pm on Jul 25, 2011 (gmt 0)

Bad bots ignore robots.txt so you need .htaccess rules to shut the door in their face.

Good bots obey robots.txt so you can keep them out of various bits of the site quite easily if those URLs have an easy to recognise pattern or feature in them.