wilderness - 3:13 am on Feb 16, 2011 (gmt 0)
4) White list Google, Yahoo and MSN in robots.txt
In your use of "white list" above, is it your intent to allow these SE's or deny these SE's?
By definition and/or practice, "white list" or "white-list" denies ALL visitors, and allows only exceptions.
Many webmasters here are using a "white-list" method in htaccess (an entirely different process than robots.txt), unfortunately there are only a few (and those offer minimal reference)examples in the Webmaster World archives. "White-listing" requires a through comprehension of browsers, UA's IP's and even other criteria, all used in a collective application to deny all and allow exceptions.
There are not any comprehensive examples of this process in the open WWW.