homepage Welcome to WebmasterWorld Guest from 54.234.141.47
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Problem of syntax and ordering
esllou

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 527 posted 11:50 am on Jan 4, 2005 (gmt 0)

here is my robots.txt. I have added to it piecemeal over the last three years and I fear it has become a bit of a mess. I find ordering hard to grasp on robots.txt. For example, will slurp come in, read the second entry concerning only it, and then dash off and ignore all the main * stuff at the bottom.

Comments appreciated...
------------
User-agent: Mediapartners-Google*
Disallow:

User-agent: Slurp
Crawl-delay: 5

User-Agent: msnbot
crawl-delay: 5

User-agent: NPBot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: googlebot
Disallow: *.cgi
Disallow: /cgi-bin/
Disallow: /phprint.php
Disallow: /*.gif$
Disallow: /*.jpg$

User-agent: *
Disallow: /graphics
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /bot-cheese.shtml
Disallow: /bot-trap.shtml
Disallow: /phprint.php
Disallow: /cgi-bin/
Disallow: /guestbook.shtml
---------------

 

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved