---- Secret AdSense Publisher ID Data Harvesting by Domain Company
koan - 6:44 am on Feb 28, 2009 (gmt 0)
they do not listen to, or even open, robots.txt files
Which is actually great if you have bot traps that block them automatically. Spam bots that follow robots.txt rules, slowly index your site from many different IP, with various user agent, and at a reasonable speed, are a lot more difficult. Did I say too much?