I use a bot trap page to block bad bots automatically when they do not obey robots.txt rules and visit it, as this page is specifically disallowed. I put an invisible 1X1 pixel image link at the bottom of my pages to this page. It's been working well for the most part. I've had my share of problems with Google Web Accelerator, which some people have called a nightmare for webmasters, and I agree, but I resigned myself to blocking it as long as it doesn't obey robots.txt rules (if it's spidering pages in advance, instead of just caching pages that the visitor visits, it SHOULD obey those rules like other crawlers, but that's another story). I figured Google dedicate a few IP addresses to Google Web Accelerator and it wouldn't hurt me with their other services if they're blocked. So far so good. And yes, the disallow rule in the robots file has been there and registered by Google for a long time.
The problem is their Sitemaps tools (now called Webmaster Tools) either share an IP address with Google Web Accelerator (64.233.172.38 is this case) or they crawl more than the verification file on your site (you know that google555555555.html file you need to upload to verify that you are the owner of a site.. yes I just made that filename up). Because when I use the Webmaster Tools, I keep having the remove that IP address from my htaccess file or the Webmaster Tools tell me their crawler has been blocked from verifying.
I know in its current state, Google Web Accelerator does not obey robots.txt rules. But I think the Webmaster Tools should!
Any idea how I can fix this?
