homepage Welcome to WebmasterWorld Guest from 23.21.9.44
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google AdSense
Forum Library, Charter, Moderators: incrediBILL & jatar k & martinibuster

Google AdSense Forum

    
Robots.txt and Mediabot
Does Mediabot respects Robots.txt?
Mentat




msg:1423103
 5:59 am on Feb 27, 2006 (gmt 0)

If I disable access for all bots (*) on ome pages in Robots.txt, does Mediabot respects this convention and on those pages I'll have redirects?

 

Hobbs




msg:1423104
 8:49 am on Feb 27, 2006 (gmt 0)

Yes it does.

AdSenseAdvisor




msg:1423105
 12:00 am on Feb 28, 2006 (gmt 0)

And if you'd like to block all robots except the AdSense crawler, check out this help page:

[google.com...]

Mentat




msg:1423106
 8:02 am on Feb 28, 2006 (gmt 0)

Thank you.
It's correct like this?


User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /users/
Disallow: /logan/

This will permit Mediapartners-Google to crawl all my site, including users and logan folders?

etgsgroup




msg:1423107
 9:52 am on Feb 28, 2006 (gmt 0)

I see many time some robot not read robots.txt
How I can do?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google AdSense
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved