Welcome to WebmasterWorld Guest from 54.196.244.206

Forum Moderators: goodroi

Message Too Old, No Replies

Is this robots.txt ok?

     
12:14 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 27, 2003
posts:835
votes: 0


any problems with syntax or ordering of commands?

______________________________

User-agent: NPBot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: googlebot
Disallow: *.cgi
Disallow: /cgi-bin/
Disallow: /phprint.php

User-agent: *
Disallow: /graphics
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /badbot.shtml
Disallow: /phprint.php
Disallow: /cgi-bin/
Diallow: /guestbook.shtml

12:16 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


Robots.txt does not support wildcards, so *.gif$ will be ignored.
12:23 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 27, 2003
posts:835
votes: 0


that's for Google, which does support wildcards

sort of a double protection against google images to go with the disallow for the image bot above.

12:27 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


Oh. Well, in that case look ok.
Check out WebmasterWorld robots.txt in case you fancy banning a few more!
12:32 pm on May 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 27, 2003
posts:835
votes: 0


thanks for your time and feedback.

I was worried about the ordering of the commands really.

I am hitting back big against bad bots and spammers on my site this week. Using .htaccess more than robots.txt but it all helps.

thanks again...

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members