| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Is this robots.txt ok?|
| 12:14 pm on May 26, 2004 (gmt 0)|
any problems with syntax or ordering of commands?
| 12:16 pm on May 26, 2004 (gmt 0)|
Robots.txt does not support wildcards, so *.gif$ will be ignored.
| 12:23 pm on May 26, 2004 (gmt 0)|
that's for Google, which does support wildcards
sort of a double protection against google images to go with the disallow for the image bot above.
| 12:27 pm on May 26, 2004 (gmt 0)|
Oh. Well, in that case look ok.
Check out WebmasterWorld robots.txt in case you fancy banning a few more!
| 12:32 pm on May 26, 2004 (gmt 0)|
thanks for your time and feedback.
I was worried about the ordering of the commands really.
I am hitting back big against bad bots and spammers on my site this week. Using .htaccess more than robots.txt but it all helps.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved