topr8 - 7:27 am on Nov 9, 2011 (gmt 0)
>>I hope that's not your only line of defense
absolutely not, however some bots as you know obey it, so i consider it worth using.
>>It's not actually a crawler, technically only crawlers need to obey robots.txt
he says on his site that he obeys robots.txt, i was just wondering if he actually did.