Welcome to WebmasterWorld Guest from 54.145.209.34

Forum Moderators: brotherhood of lan & mack

Help with robots.txt for Wordpress blog

   
12:35 pm on Dec 16, 2008 (gmt 0)

5+ Year Member



Hi there!

I'm new to managing a self hosted blog, and found myself very lost with this robots.txt file.

I've been lurking the net for some directions, but found different options everywhere (specially for Wordpress), so I don't know what to choose - also, I've seen a lot of those robots.txt articles filled with some comments saying that "it doesn't work' or 'it messed up my blog' - so I grew scared.

What I want/need is a simple robots.txt file able to:

Avoid Google Image search crawling my images.
Avoid duplicate content
Allow the Adsense bot to run freely in my blog :)

Would you help me with this? It will be much appreciated.

Thanks for any imput, and please bear my English :)

[edited by: Oxydada at 12:36 pm (utc) on Dec. 16, 2008]

7:04 am on Dec 17, 2008 (gmt 0)

5+ Year Member



Avoid Google Image search crawling images :
The traffic from the Image search engine is very low quality and rarely converts into buyers. Many people are often just looking for images that they can swipe. So, if you want to save some bandwidth, use your robots.txt file to block ImageBot from accessing your image directory.

useragent: GoogleBot-Image
Disallow: /images/

or you can easily prevent Google from indexing your pictures by placing the following code into your blog’s header file above the < /head > tag:

<meta name="robots" content="noimageindex">

---------------------------------------------------------

Allow the Adsense bot :
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to determine wich ads Google should display on Adsense pages.

Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site.

User-agent: Mediapartners-Google*
Disallow:
Allow: /*

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month