I'm new to managing a self hosted blog, and found myself very lost with this robots.txt file.
I've been lurking the net for some directions, but found different options everywhere (specially for Wordpress), so I don't know what to choose - also, I've seen a lot of those robots.txt articles filled with some comments saying that "it doesn't work' or 'it messed up my blog' - so I grew scared.
What I want/need is a simple robots.txt file able to:
Avoid Google Image search crawling my images.
Avoid duplicate content
Allow the Adsense bot to run freely in my blog :)
Would you help me with this? It will be much appreciated.
Thanks for any imput, and please bear my English :)
[edited by: Oxydada at 12:36 pm (utc) on Dec. 16, 2008]