Welcome to WebmasterWorld Guest from

Forum Moderators: keyplyr & mack

Message Too Old, No Replies

Help with robots.txt for Wordpress blog

12:35 pm on Dec 16, 2008 (gmt 0)

Junior Member

5+ Year Member

joined:June 4, 2008
posts: 66
votes: 0

Hi there!

I'm new to managing a self hosted blog, and found myself very lost with this robots.txt file.

I've been lurking the net for some directions, but found different options everywhere (specially for Wordpress), so I don't know what to choose - also, I've seen a lot of those robots.txt articles filled with some comments saying that "it doesn't work' or 'it messed up my blog' - so I grew scared.

What I want/need is a simple robots.txt file able to:

Avoid Google Image search crawling my images.
Avoid duplicate content
Allow the Adsense bot to run freely in my blog :)

Would you help me with this? It will be much appreciated.

Thanks for any imput, and please bear my English :)

[edited by: Oxydada at 12:36 pm (utc) on Dec. 16, 2008]

7:04 am on Dec 17, 2008 (gmt 0)

New User

5+ Year Member

joined:Dec 16, 2008
votes: 0

Avoid Google Image search crawling images :
The traffic from the Image search engine is very low quality and rarely converts into buyers. Many people are often just looking for images that they can swipe. So, if you want to save some bandwidth, use your robots.txt file to block ImageBot from accessing your image directory.

useragent: GoogleBot-Image
Disallow: /images/

or you can easily prevent Google from indexing your pictures by placing the following code into your blog’s header file above the < /head > tag:

<meta name="robots" content="noimageindex">


Allow the Adsense bot :
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to determine wich ads Google should display on Adsense pages.

Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site.

User-agent: Mediapartners-Google*
Allow: /*


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members