Forum Moderators: open
I have recently observed my log file and saw lot's of visit fromMediapartners-Google/2.1 in my index page. Though I don't have google adsence ad in my Website.It is consuming lots of my valuable bandwidth each day by giving 1000s of visits of the same index file. and it is not indexing my page in google. So I want to disallow it to crawl my site. BUt I want googlebot to crawl my site.
I m using php-nuke mod_rewrite for search engine optimization.
I have also observed Mediapartners-Google/2.1barely call for my robots.txt . What is the Mediapartners-Google/2.1 's robot exclusion name for robots.txt file? What should I put in my robot.txt file to disallow Mediapartners-Google/2.1 not Google.
Thanks in advance.
"I tested it in on a new site without Adsense. The mediapartners bot follows the opera browser around the site within the minute."
[webmasterworld.com...]
I know the robot exclusion tag for google which is google for yahoo it is slurp but what would be for Mediapartners? I tried
http://www.robotstxt.org/wc/active/html/
Any hope?