supercyberbob - 12:53 am on May 31, 2010 (gmt 0)

pageoneresults

I use the noindex meta tag on some of my sites instead of robots.txt entries. I agree robots.txt is like having the curtains open.

I haven't examined crawling on those sites though. Might be a good idea at some point.