homepage Welcome to WebmasterWorld Guest from 54.211.47.170
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Expert opinion on this robots.txt for wordpress blog
robots.txt review
denharsh




msg:3881383
 11:53 pm on Mar 29, 2009 (gmt 0)

Hey
This is how my robots.txt file look like

sitemap: http://www.example.com/sitemap.xml

User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /stats/
Disallow: /dh_
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /contact/
Disallow: /trackback/
Disallow: /category/
Disallow: /tag/
Disallow: /author/
Disallow: /go/
Disallow: /page/
Disallow: /wp-images/
Disallow: /images/
Disallow: /banners/
Disallow: /archives/
Disallow: /feed/
Disallow: /*?*
Disallow: */trackback/
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /backup/
Disallow: /*/feed/
Disallow: /*/trackback/
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: */trackback*
Disallow: /wp-*
Disallow: /ar/
Disallow: /bg/
Disallow: /ca/
Disallow: /cs/
Disallow: /da/
Disallow: /de/
Disallow: /el/
Disallow: /es/
Disallow: /fi/
Disallow: /fr/
Disallow: /hi/
Disallow: /hr/
Disallow: /id/
Disallow: /it/
Disallow: /iw/
Disallow: /ja/
Disallow: /ko/
Disallow: /lt/
Disallow: /lv/
Disallow: /nl/
Disallow: /no/
Disallow: /pl/
Disallow: /pt/
Disallow: /ro/
Disallow: /ru/
Disallow: /sk/
Disallow: /sl/
Disallow: /sr/
Disallow: /sv/
Disallow: /tl/
Disallow: /uk/
Disallow: /vi/
Disallow: /zh-CN/

User-agent: Mediapartners-Google
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

Am I doing anything wrong here? or is it perfect?
Any Idea or suggestion will be much appreciated....

[edited by: eelixduppy at 12:11 am (utc) on Mar. 30, 2009]
[edit reason] exemplified [/edit]

 

phranque




msg:3881406
 12:32 am on Mar 30, 2009 (gmt 0)

welcome to WebmasterWorld [webmasterworld.com], denharsh!

Mediapartners-Google will be able to crawl everything.

some of your directive are redundant. for example, the following directive:
Disallow: /wp-*
also covers these directives:
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /wp-images/

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved