Welcome to WebmasterWorld Guest from 54.167.40.25

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

Robots.txt and Mediabot

Does Mediabot respects Robots.txt?

     
5:59 am on Feb 27, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Nov 6, 2003
posts:1058
votes: 13


If I disable access for all bots (*) on ome pages in Robots.txt, does Mediabot respects this convention and on those pages I'll have redirects?
8:49 am on Feb 27, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 19, 2004
posts:3054
votes: 3


Yes it does.
12:00 am on Feb 28, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 4, 2003
posts:1161
votes: 0


And if you'd like to block all robots except the AdSense crawler, check out this help page:

[google.com...]

8:02 am on Feb 28, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Nov 6, 2003
posts:1058
votes: 13


Thank you.
It's correct like this?


User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /users/
Disallow: /logan/

This will permit Mediapartners-Google to crawl all my site, including users and logan folders?

9:52 am on Feb 28, 2006 (gmt 0)

New User

10+ Year Member

joined:Dec 30, 2004
posts:28
votes: 0


I see many time some robot not read robots.txt
How I can do?