Welcome to WebmasterWorld Guest from 54.167.5.15

Forum Moderators: goodroi

Message Too Old, No Replies

Is this robots.txt ok?

     

esllou

12:14 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



any problems with syntax or ordering of commands?

______________________________

User-agent: NPBot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: googlebot
Disallow: *.cgi
Disallow: /cgi-bin/
Disallow: /phprint.php

User-agent: *
Disallow: /graphics
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /badbot.shtml
Disallow: /phprint.php
Disallow: /cgi-bin/
Diallow: /guestbook.shtml

Sanenet

12:16 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Robots.txt does not support wildcards, so *.gif$ will be ignored.

esllou

12:23 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



that's for Google, which does support wildcards

sort of a double protection against google images to go with the disallow for the image bot above.

Sanenet

12:27 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Oh. Well, in that case look ok.
Check out WebmasterWorld robots.txt in case you fancy banning a few more!

esllou

12:32 pm on May 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



thanks for your time and feedback.

I was worried about the ordering of the commands really.

I am hitting back big against bad bots and spammers on my site this week. Using .htaccess more than robots.txt but it all helps.

thanks again...

 

Featured Threads

Hot Threads This Week

Hot Threads This Month