Welcome to WebmasterWorld Guest from 54.167.82.170

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

Robots.txt and Mediabot

Does Mediabot respects Robots.txt?

     

Mentat

5:59 am on Feb 27, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If I disable access for all bots (*) on ome pages in Robots.txt, does Mediabot respects this convention and on those pages I'll have redirects?

Hobbs

8:49 am on Feb 27, 2006 (gmt 0)

WebmasterWorld Senior Member hobbs is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Yes it does.

AdSenseAdvisor

12:00 am on Feb 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



And if you'd like to block all robots except the AdSense crawler, check out this help page:

[google.com...]

Mentat

8:02 am on Feb 28, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thank you.
It's correct like this?


User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /users/
Disallow: /logan/

This will permit Mediapartners-Google to crawl all my site, including users and logan folders?

etgsgroup

9:52 am on Feb 28, 2006 (gmt 0)

10+ Year Member



I see many time some robot not read robots.txt
How I can do?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month