Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
Problem of syntax and ordering esllou msg:1527111 11:50 am on Jan 4, 2005 (gmt 0) here is my robots.txt. I have added to it piecemeal over the last three years and I fear it has become a bit of a mess. I find ordering hard to grasp on robots.txt. For example, will slurp come in, read the second entry concerning only it, and then dash off and ignore all the main * stuff at the bottom.
------------ User-agent: Mediapartners-Google* Disallow:
Disallow: *.cgi Disallow: /cgi-bin/ Disallow: /phprint.php Disallow: /*.gif$ Disallow: /*.jpg$
Disallow: /graphics Disallow: /*.gif$ Disallow: /*.jpg$ Disallow: /bot-cheese.shtml Disallow: /bot-trap.shtml Disallow: /phprint.php Disallow: /cgi-bin/ Disallow: /guestbook.shtml ---------------