| Welcome to WebmasterWorld Guest from 188.8.131.52 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Robots.txt and Mediabot|
Does Mediabot respects Robots.txt?
| 5:59 am on Feb 27, 2006 (gmt 0)|
If I disable access for all bots (*) on ome pages in Robots.txt, does Mediabot respects this convention and on those pages I'll have redirects?
| 8:49 am on Feb 27, 2006 (gmt 0)|
Yes it does.
| 12:00 am on Feb 28, 2006 (gmt 0)|
And if you'd like to block all robots except the AdSense crawler, check out this help page:
| 8:02 am on Feb 28, 2006 (gmt 0)|
It's correct like this?
This will permit Mediapartners-Google to crawl all my site, including users and logan folders?
| 9:52 am on Feb 28, 2006 (gmt 0)|
I see many time some robot not read robots.txt
How I can do?
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved