Welcome to WebmasterWorld Guest from 54.161.187.250

Forum Moderators: goodroi

Message Too Old, No Replies

Do these robot.txt's look ok?

   
10:26 pm on Feb 13, 2010 (gmt 0)

5+ Year Member



They are for separate URL's. The first is for my hobby wordpress blog:


User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /z/j/
Disallow: /z/c/
Disallow: /stats/
Disallow: /dh_
Disallow: /about/
Disallow: /contact/
Disallow: /tag/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /contact
Disallow: /manual
Disallow: /manual/*
Disallow: /phpmanual/
Disallow: /category/

The second is for my commercial static site that also has a forum attached(simple Machine Forum software):

User-agent: *
Disallow:

Disallow: /forums/index.php?action=help
Disallow: /forums/index.php?action=search
Disallow: /forums/index.php?action=login
Disallow: /forums/index.php?action=register
Disallow: /forums/index.php?action=admin
Disallow: /forums/index.php?action=post
Disallow: /forums/index.php?action=who
disallow: /forums/index.php?action=printpage



Thanks for any feedback.
9:24 am on Mar 16, 2010 (gmt 0)

5+ Year Member



Hi........ Yes it is ok
1:29 pm on Mar 16, 2010 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



it depends on what you are looking for.
robots.tx Disallow prevents crawling but it doesn't prevent indexing.
therefore these directives will prevent the content of those pages from being crawled but they won't prevent the urls from being indexed.
that requires a robots.txt that allows crawling and a robots noindex metatag in the document or the appropriate X-Robots-Tag http header.
1:45 pm on Mar 16, 2010 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



i would also suggest you check out the interview with Matt Cutts by Eric Enge [webmasterworld.com] and discussion on this subject.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month