homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

Help with robots.txt for Wordpress blog

5+ Year Member

Msg#: 3808520 posted 12:35 pm on Dec 16, 2008 (gmt 0)

Hi there!

I'm new to managing a self hosted blog, and found myself very lost with this robots.txt file.

I've been lurking the net for some directions, but found different options everywhere (specially for Wordpress), so I don't know what to choose - also, I've seen a lot of those robots.txt articles filled with some comments saying that "it doesn't work' or 'it messed up my blog' - so I grew scared.

What I want/need is a simple robots.txt file able to:

Avoid Google Image search crawling my images.
Avoid duplicate content
Allow the Adsense bot to run freely in my blog :)

Would you help me with this? It will be much appreciated.

Thanks for any imput, and please bear my English :)

[edited by: Oxydada at 12:36 pm (utc) on Dec. 16, 2008]



5+ Year Member

Msg#: 3808520 posted 7:04 am on Dec 17, 2008 (gmt 0)

Avoid Google Image search crawling images :
The traffic from the Image search engine is very low quality and rarely converts into buyers. Many people are often just looking for images that they can swipe. So, if you want to save some bandwidth, use your robots.txt file to block ImageBot from accessing your image directory.

useragent: GoogleBot-Image
Disallow: /images/

or you can easily prevent Google from indexing your pictures by placing the following code into your blog’s header file above the < /head > tag:

<meta name="robots" content="noimageindex">


Allow the Adsense bot :
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to determine wich ads Google should display on Adsense pages.

Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site.

User-agent: Mediapartners-Google*
Allow: /*

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved