| Welcome to WebmasterWorld Guest from 188.8.131.52 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
|Is this robots.txt ok?|
| 12:14 pm on May 26, 2004 (gmt 0)|
any problems with syntax or ordering of commands?
| 12:16 pm on May 26, 2004 (gmt 0)|
Robots.txt does not support wildcards, so *.gif$ will be ignored.
| 12:23 pm on May 26, 2004 (gmt 0)|
that's for Google, which does support wildcards
sort of a double protection against google images to go with the disallow for the image bot above.
| 12:27 pm on May 26, 2004 (gmt 0)|
Oh. Well, in that case look ok.
Check out WebmasterWorld robots.txt in case you fancy banning a few more!
| 12:32 pm on May 26, 2004 (gmt 0)|
thanks for your time and feedback.
I was worried about the ordering of the commands really.
I am hitting back big against bad bots and spammers on my site this week. Using .htaccess more than robots.txt but it all helps.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2014 all rights reserved