bigtoga - 1:22 pm on May 13, 2013 (gmt 0)
Good info - thank you. I'm definitely in the "ordinary mortals" category.
My thinking behind this was that having a solidified robots.txt that says "Only these bots can do these specific things", to me, makes sense for justifying a super fast banning policy. I can look at my morning reports, find anyone scraping, and poof - you're out. I like having the "backup" that says, "I see you crawling, I've said you can't crawl, so now you are banned." And I say that to myself haha - I know "they" aren't listening. I was thinking that it just makes more sense and makes it easier for me.