Welcome to WebmasterWorld Guest from 184.73.112.180

Forum Moderators: goodroi

Expert opinion on this robots.txt for wordpress blog

robots.txt review

   
11:53 pm on Mar 29, 2009 (gmt 0)

5+ Year Member



Hey
This is how my robots.txt file look like

sitemap: http://www.example.com/sitemap.xml

User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /stats/
Disallow: /dh_
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /contact/
Disallow: /trackback/
Disallow: /category/
Disallow: /tag/
Disallow: /author/
Disallow: /go/
Disallow: /page/
Disallow: /wp-images/
Disallow: /images/
Disallow: /banners/
Disallow: /archives/
Disallow: /feed/
Disallow: /*?*
Disallow: */trackback/
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /backup/
Disallow: /*/feed/
Disallow: /*/trackback/
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: */trackback*
Disallow: /wp-*
Disallow: /ar/
Disallow: /bg/
Disallow: /ca/
Disallow: /cs/
Disallow: /da/
Disallow: /de/
Disallow: /el/
Disallow: /es/
Disallow: /fi/
Disallow: /fr/
Disallow: /hi/
Disallow: /hr/
Disallow: /id/
Disallow: /it/
Disallow: /iw/
Disallow: /ja/
Disallow: /ko/
Disallow: /lt/
Disallow: /lv/
Disallow: /nl/
Disallow: /no/
Disallow: /pl/
Disallow: /pt/
Disallow: /ro/
Disallow: /ru/
Disallow: /sk/
Disallow: /sl/
Disallow: /sr/
Disallow: /sv/
Disallow: /tl/
Disallow: /uk/
Disallow: /vi/
Disallow: /zh-CN/

User-agent: Mediapartners-Google
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

Am I doing anything wrong here? or is it perfect?
Any Idea or suggestion will be much appreciated....

[edited by: eelixduppy at 12:11 am (utc) on Mar. 30, 2009]
[edit reason] exemplified [/edit]

12:32 am on Mar 30, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



welcome to WebmasterWorld [webmasterworld.com], denharsh!

Mediapartners-Google will be able to crawl everything.

some of your directive are redundant. for example, the following directive:
Disallow: /wp-*
also covers these directives:
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /wp-images/

 

Featured Threads

Hot Threads This Week

Hot Threads This Month