incrediBILL - 9:27 am on Feb 25, 2012 (gmt 0)

Of course, the crawlers respect robots.txt and will abide if you don't want them around.

Of course, unless your robots.txt is set up in whitelisting format, which virtually nobody does, they don't know who you are initially so there is no disallow to honor, ever.