Forum Moderators: open

Message Too Old, No Replies

How to disallow Mediapartners

Need help on robot.txt

         

coolsaint

5:44 pm on Jan 15, 2005 (gmt 0)

10+ Year Member



Dear friends,

I have recently observed my log file and saw lot's of visit fromMediapartners-Google/2.1 in my index page. Though I don't have google adsence ad in my Website.It is consuming lots of my valuable bandwidth each day by giving 1000s of visits of the same index file. and it is not indexing my page in google. So I want to disallow it to crawl my site. BUt I want googlebot to crawl my site.
I m using php-nuke mod_rewrite for search engine optimization.

I have also observed Mediapartners-Google/2.1barely call for my robots.txt . What is the Mediapartners-Google/2.1 's robot exclusion name for robots.txt file? What should I put in my robot.txt file to disallow Mediapartners-Google/2.1 not Google.

Thanks in advance.

fiestagirl

8:19 pm on Jan 17, 2005 (gmt 0)

10+ Year Member



Previous thread, possibly related.

"I tested it in on a new site without Adsense. The mediapartners bot follows the opera browser around the site within the minute."

[webmasterworld.com...]

coolsaint

2:27 am on Jan 18, 2005 (gmt 0)

10+ Year Member



I read the topic before posting a new topic. I did a search for Mediapartners in the forum.
I know how they come from opera browser. But I want to disallow them. I wanted to know how.

I know the robot exclusion tag for google which is google for yahoo it is slurp but what would be for Mediapartners? I tried

http://www.robotstxt.org/wc/active/html/

But it is not in that list. I can't find the exclusion tag for mediapartners of google in that list. It is not listed.

Any hope?

jdMorgan

3:40 am on Jan 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I haven't had this problem, but I'd suggest the following.

In robots.txt:

User-agent: Mediapartners-Google
Disallow: /

And in .htaccess:


RewriteCond %{HTTP_USER_AGENT} Mediapartners-Google
RewriteCond $1 !^robots\.txt$
RewriteCond $1 !^Your_custom_403_page\.php$
RewriteRule (.*) - [F]

Jim