Welcome to WebmasterWorld Guest from 54.159.246.164

Forum Moderators: goodroi

Message Too Old, No Replies

Is this robots.txt ok?

   
12:14 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



any problems with syntax or ordering of commands?

______________________________

User-agent: NPBot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: googlebot
Disallow: *.cgi
Disallow: /cgi-bin/
Disallow: /phprint.php

User-agent: *
Disallow: /graphics
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /badbot.shtml
Disallow: /phprint.php
Disallow: /cgi-bin/
Diallow: /guestbook.shtml

12:16 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Robots.txt does not support wildcards, so *.gif$ will be ignored.
12:23 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



that's for Google, which does support wildcards

sort of a double protection against google images to go with the disallow for the image bot above.

12:27 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Oh. Well, in that case look ok.
Check out WebmasterWorld robots.txt in case you fancy banning a few more!
12:32 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



thanks for your time and feedback.

I was worried about the ordering of the commands really.

I am hitting back big against bad bots and spammers on my site this week. Using .htaccess more than robots.txt but it all helps.

thanks again...