Mediabot is requesting a page 1000's of times a day and this page NEVER had any ads on it. The user who originally went to this page was using a "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/85 (KHTML, like Gecko) Safari/85"
Of the 30,000 messages in my mail section this is the only one being requested. What do i do?
Can you ban it by using robots.txt for the /message.asp? You can just ban the AdSense bot, and not the Googlebot, if that is a concern.
I know that the AdSense bot is requesting robots.txt nearly everytime it visits any of my sites, so it should solve the problem quickly.
Here is the information for the AdSense bot using robots.txt https://www.google.com/adsense/faq-tech#q6 Note that the information on the page is to ALLOW the AdSense bot on all pages, so you will need to change it accordingly, and you should probably run it through the robots.txt validator here before you unleash it ;)