homepage Welcome to WebmasterWorld Guest from 107.22.156.205
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Visit PubCon.com
Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library : Charter : Moderators: brotherhood of lan & mack

New To Web Development Forum

    
Help with robots.txt for Wordpress blog
Oxydada




msg:3808522		 12:35 pm on Dec 16, 2008 (gmt 0)

Hi there!

I'm new to managing a self hosted blog, and found myself very lost with this robots.txt file.

I've been lurking the net for some directions, but found different options everywhere (specially for Wordpress), so I don't know what to choose - also, I've seen a lot of those robots.txt articles filled with some comments saying that "it doesn't work' or 'it messed up my blog' - so I grew scared.

What I want/need is a simple robots.txt file able to:

Avoid Google Image search crawling my images.
Avoid duplicate content
Allow the Adsense bot to run freely in my blog :)

Would you help me with this? It will be much appreciated.

Thanks for any imput, and please bear my English :)

[edited by: Oxydada at 12:36 pm (utc) on Dec. 16, 2008]

 

gibbs_h




msg:3809297		 7:04 am on Dec 17, 2008 (gmt 0)

Avoid Google Image search crawling images :
The traffic from the Image search engine is very low quality and rarely converts into buyers. Many people are often just looking for images that they can swipe. So, if you want to save some bandwidth, use your robots.txt file to block ImageBot from accessing your image directory.

useragent: GoogleBot-Image
Disallow: /images/

or you can easily prevent Google from indexing your pictures by placing the following code into your blog’s header file above the < /head > tag:

<meta name="robots" content="noimageindex">

---------------------------------------------------------

Allow the Adsense bot :
MediaBot is the Google crawler for Adsense Publishers. Mediabot is used to determine wich ads Google should display on Adsense pages.

Google recommends that webmasters specifically add a command in their robots.txt file that grants Mediabot access to their entire site.

User-agent: Mediapartners-Google*
Disallow:
Allow: /*

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved