If the attacker wanted to attack you, even you shutdown your site and web server, they can just copy your Adsense ID and display ads on the attacker's own scraper site to abuse it.
That's a really good remark, Burntan. For a long time I am trying to convince Google to tie publisher's code to the actually registered publisher's sites. In that way only the clicks coming from your site and matching your code would be counted and credited. Not only because of the reason you mentioned, but also to wipe out now huge businesses using stealware/parasites on users' pc's in order to steal comissions.