incrediBILL - 8:46 pm on Dec 27, 2012 (gmt 0)

Bad bots never even look at robots.txt

Actually they do at times. There's one bot that uses a blank user agent when reading the robots.txt to see which spider names you've allowed. Then it switches to one of those spider names to make sure it gets access to your site.

Another reason I do a dynamic robots.txt file that tells everyone to go away except my whitelist and serves up a custom file per request so that I don't expose the bot names that are allowed.

In case you're wondering, ia_archive is NOT allowed ;)