I am having this problem also for one of the sites I manage. Unlike my other sites, which I host from my own server, this one site is hosted at <unnamed> web service provider. Evidently, they are doing something funky with the domain name, directory structure, or whatever.
The Google SiteMap page has complained about not finding the sitemap file, robots.txt, and the requested verification file.
I have tried using several different variants of the domain name for this site (www vs. non-www vs. other variants besides), but nothing so far has worked. Probably, I will have to contact tech support.
Hmm, this is a clue why it appears that the Adsense mediabot is not visiting most (but not all) pages, and why this site continues to have untargeted, generic ads based on the page URLs but not page content.