Forum Moderators: mack
I'm new to managing a self hosted blog, and found myself very lost with this robots.txt file.
I've been lurking the net for some directions, but found different options everywhere (specially for Wordpress), so I don't know what to choose - also, I've seen a lot of those robots.txt articles filled with some comments saying that "it doesn't work' or 'it messed up my blog' - so I grew scared.
What I want/need is a simple robots.txt file able to:
Avoid Google Image search crawling my images.
Avoid duplicate content
Allow the Adsense bot to run freely in my blog :)
Would you help me with this? It will be much appreciated.
Thanks for any imput, and please bear my English :)
[edited by: Oxydada at 12:36 pm (utc) on Dec. 16, 2008]
useragent: GoogleBot-Image
Disallow: /images/
or you can easily prevent Google from indexing your pictures by placing the following code into your blog’s header file above the < /head > tag:
<meta name="robots" content="noimageindex">
---------------------------------------------------------
Allow the Adsense bot :
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to determine wich ads Google should display on Adsense pages.
Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site.
User-agent: Mediapartners-Google*
Disallow:
Allow: /*