topr8 - 10:17 am on Dec 30, 2010 (gmt 0) @koan you need a dynamic robots txt - the default is Block all then test for the spiders you want and serve them a different set of rules
@koan you need a dynamic robots txt - the default is Block all then test for the spiders you want and serve them a different set of rules