Welcome to WebmasterWorld Guest from 54.156.92.140

Forum Moderators: goodroi

Message Too Old, No Replies

Is this robots.txt ok?

     
12:14 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 27, 2003
posts:835
votes: 0


any problems with syntax or ordering of commands?

______________________________

User-agent: NPBot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: googlebot
Disallow: *.cgi
Disallow: /cgi-bin/
Disallow: /phprint.php

User-agent: *
Disallow: /graphics
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /badbot.shtml
Disallow: /phprint.php
Disallow: /cgi-bin/
Diallow: /guestbook.shtml

12:16 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


Robots.txt does not support wildcards, so *.gif$ will be ignored.
12:23 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 27, 2003
posts:835
votes: 0


that's for Google, which does support wildcards

sort of a double protection against google images to go with the disallow for the image bot above.

12:27 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


Oh. Well, in that case look ok.
Check out WebmasterWorld robots.txt in case you fancy banning a few more!
12:32 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 27, 2003
posts:835
votes: 0


thanks for your time and feedback.

I was worried about the ordering of the commands really.

I am hitting back big against bad bots and spammers on my site this week. Using .htaccess more than robots.txt but it all helps.

thanks again...