homepage Welcome to WebmasterWorld Guest from 50.17.86.12
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google AdSense
Forum Library, Charter, Moderators: incrediBILL & jatar k & martinibuster

Google AdSense Forum

    
Robots.txt and Mediabot
Does Mediabot respects Robots.txt?
Mentat

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 12347 posted 5:59 am on Feb 27, 2006 (gmt 0)

If I disable access for all bots (*) on ome pages in Robots.txt, does Mediabot respects this convention and on those pages I'll have redirects?

 

Hobbs

WebmasterWorld Senior Member hobbs us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 12347 posted 8:49 am on Feb 27, 2006 (gmt 0)

Yes it does.

AdSenseAdvisor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 12347 posted 12:00 am on Feb 28, 2006 (gmt 0)

And if you'd like to block all robots except the AdSense crawler, check out this help page:

[google.com...]

Mentat

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 12347 posted 8:02 am on Feb 28, 2006 (gmt 0)

Thank you.
It's correct like this?


User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /users/
Disallow: /logan/

This will permit Mediapartners-Google to crawl all my site, including users and logan folders?

etgsgroup

5+ Year Member



 
Msg#: 12347 posted 9:52 am on Feb 28, 2006 (gmt 0)

I see many time some robot not read robots.txt
How I can do?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google AdSense
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved