robzilla - 12:43 pm on Feb 27, 2009 (gmt 0) Correction: it looks like they have a 'sitemap' of search result pages, so it appears they are at least trying to get all the IDs indexed by search engines.
Clever trick. I'm not sure what's worse, this particular site that's fully indexed by Google, or another similar service (in English) that actually sells this information and has a much bigger database but isn't publicly available (i.e. the results aren't indexed, you have to pay first). The latter, which is quite easy to find, seems to be relatively popular and has over 700k sites in its index, whereas the German site claims 200k. I tried a few Adsense IDs, without paying, and they seemed to provide pretty complete results (they show you how many domains and subdomains use the ID and then make you pay to see them). This allows anyone to see if competitors are doing anything shady like arbitrage. A new can of "outing" worms. Additionally, the owner claims that what they are doing is perfectly legal and says they do not listen to, or even open, robots.txt files. Any blocking of the robot would have to happen on the basis of IP(s), but for some reason I doubt they are crawling with a user-agent that will easily identify them.
Correction: it looks like they have a 'sitemap' of search result pages, so it appears they are at least trying to get all the IDs indexed by search engines.