Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor 2014
Problem of syntax and ordering esllou
here is my robots.txt. I have added to it piecemeal over the last three years and I fear it has become a bit of a mess. I find ordering hard to grasp on robots.txt. For example, will slurp come in, read the second entry concerning only it, and then dash off and ignore all the main * stuff at the bottom.
------------ User-agent: Mediapartners-Google* Disallow:
Disallow: *.cgi Disallow: /cgi-bin/ Disallow: /phprint.php Disallow: /*.gif$ Disallow: /*.jpg$
Disallow: /graphics Disallow: /*.gif$ Disallow: /*.jpg$ Disallow: /bot-cheese.shtml Disallow: /bot-trap.shtml Disallow: /phprint.php Disallow: /cgi-bin/ Disallow: /guestbook.shtml ---------------