homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google AdSense
Forum Library, Charter, Moderators: incrediBILL & jatar k & martinibuster

Google AdSense Forum

Robots.txt and Mediabot
Does Mediabot respects Robots.txt?

 5:59 am on Feb 27, 2006 (gmt 0)

If I disable access for all bots (*) on ome pages in Robots.txt, does Mediabot respects this convention and on those pages I'll have redirects?



 8:49 am on Feb 27, 2006 (gmt 0)

Yes it does.


 12:00 am on Feb 28, 2006 (gmt 0)

And if you'd like to block all robots except the AdSense crawler, check out this help page:



 8:02 am on Feb 28, 2006 (gmt 0)

Thank you.
It's correct like this?

User-agent: Mediapartners-Google*

User-agent: *
Disallow: /users/
Disallow: /logan/

This will permit Mediapartners-Google to crawl all my site, including users and logan folders?


 9:52 am on Feb 28, 2006 (gmt 0)

I see many time some robot not read robots.txt
How I can do?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google AdSense
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved