ergophobe - 3:43 pm on Jul 12, 2013 (gmt 0)
>>tell the bots not to visit/index
lorax - robots.txt does not tell SEs not to index a site. Even worse, if the site is indexed and then you block the crawl with robots.txt, you at that point can't add a rel=noindex meta tag.
If you don't want a page/site indexed, use a meta tag with rel=noindex. Note that the page will still get crawled.
If you don't want a page/site crawled, use robots.txt. Note that the page may still get indexed.
If you don't want a page crawled or indexed, use both.