Welcome to WebmasterWorld Guest from 54.227.110.209

Forum Moderators: goodroi

Message Too Old, No Replies

Problem of syntax and ordering

     

esllou

11:50 am on Jan 4, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



here is my robots.txt. I have added to it piecemeal over the last three years and I fear it has become a bit of a mess. I find ordering hard to grasp on robots.txt. For example, will slurp come in, read the second entry concerning only it, and then dash off and ignore all the main * stuff at the bottom.

Comments appreciated...
------------
User-agent: Mediapartners-Google*
Disallow:

User-agent: Slurp
Crawl-delay: 5

User-Agent: msnbot
crawl-delay: 5

User-agent: NPBot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: googlebot
Disallow: *.cgi
Disallow: /cgi-bin/
Disallow: /phprint.php
Disallow: /*.gif$
Disallow: /*.jpg$

User-agent: *
Disallow: /graphics
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /bot-cheese.shtml
Disallow: /bot-trap.shtml
Disallow: /phprint.php
Disallow: /cgi-bin/
Disallow: /guestbook.shtml
---------------

 

Featured Threads

Hot Threads This Week

Hot Threads This Month