Welcome to WebmasterWorld Guest from 54.205.75.60

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

Robots.txt and Mediabot

Does Mediabot respects Robots.txt?

   
5:59 am on Feb 27, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If I disable access for all bots (*) on ome pages in Robots.txt, does Mediabot respects this convention and on those pages I'll have redirects?
8:49 am on Feb 27, 2006 (gmt 0)

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Yes it does.
12:00 am on Feb 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



And if you'd like to block all robots except the AdSense crawler, check out this help page:

[google.com...]

8:02 am on Feb 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thank you.
It's correct like this?


User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /users/
Disallow: /logan/

This will permit Mediapartners-Google to crawl all my site, including users and logan folders?

9:52 am on Feb 28, 2006 (gmt 0)

10+ Year Member



I see many time some robot not read robots.txt
How I can do?