homepage Welcome to WebmasterWorld Guest from 54.196.120.58
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Do these robot.txt's look ok?
Boulder90




msg:4079783
 10:26 pm on Feb 13, 2010 (gmt 0)

They are for separate URL's. The first is for my hobby wordpress blog:


User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /z/j/
Disallow: /z/c/
Disallow: /stats/
Disallow: /dh_
Disallow: /about/
Disallow: /contact/
Disallow: /tag/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /contact
Disallow: /manual
Disallow: /manual/*
Disallow: /phpmanual/
Disallow: /category/

The second is for my commercial static site that also has a forum attached(simple Machine Forum software):

User-agent: *
Disallow:

Disallow: /forums/index.php?action=help
Disallow: /forums/index.php?action=search
Disallow: /forums/index.php?action=login
Disallow: /forums/index.php?action=register
Disallow: /forums/index.php?action=admin
Disallow: /forums/index.php?action=post
Disallow: /forums/index.php?action=who
disallow: /forums/index.php?action=printpage



Thanks for any feedback.

 

ricardo12




msg:4098643
 9:24 am on Mar 16, 2010 (gmt 0)

Hi........ Yes it is ok

phranque




msg:4098734
 1:29 pm on Mar 16, 2010 (gmt 0)

it depends on what you are looking for.
robots.tx Disallow prevents crawling but it doesn't prevent indexing.
therefore these directives will prevent the content of those pages from being crawled but they won't prevent the urls from being indexed.
that requires a robots.txt that allows crawling and a robots noindex metatag in the document or the appropriate X-Robots-Tag http header.

phranque




msg:4098751
 1:45 pm on Mar 16, 2010 (gmt 0)

i would also suggest you check out the interview with Matt Cutts by Eric Enge [webmasterworld.com] and discussion on this subject.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved