homepage Welcome to WebmasterWorld Guest from 54.211.73.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Do these robot.txt's look ok?
Boulder90

5+ Year Member



 
Msg#: 4079797 posted 10:26 pm on Feb 13, 2010 (gmt 0)

They are for separate URL's. The first is for my hobby wordpress blog:


User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /z/j/
Disallow: /z/c/
Disallow: /stats/
Disallow: /dh_
Disallow: /about/
Disallow: /contact/
Disallow: /tag/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /contact
Disallow: /manual
Disallow: /manual/*
Disallow: /phpmanual/
Disallow: /category/

The second is for my commercial static site that also has a forum attached(simple Machine Forum software):

User-agent: *
Disallow:

Disallow: /forums/index.php?action=help
Disallow: /forums/index.php?action=search
Disallow: /forums/index.php?action=login
Disallow: /forums/index.php?action=register
Disallow: /forums/index.php?action=admin
Disallow: /forums/index.php?action=post
Disallow: /forums/index.php?action=who
disallow: /forums/index.php?action=printpage



Thanks for any feedback.

 

ricardo12



 
Msg#: 4079797 posted 9:24 am on Mar 16, 2010 (gmt 0)

Hi........ Yes it is ok

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4079797 posted 1:29 pm on Mar 16, 2010 (gmt 0)

it depends on what you are looking for.
robots.tx Disallow prevents crawling but it doesn't prevent indexing.
therefore these directives will prevent the content of those pages from being crawled but they won't prevent the urls from being indexed.
that requires a robots.txt that allows crawling and a robots noindex metatag in the document or the appropriate X-Robots-Tag http header.

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4079797 posted 1:45 pm on Mar 16, 2010 (gmt 0)

i would also suggest you check out the interview with Matt Cutts by Eric Enge [webmasterworld.com] and discussion on this subject.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved