| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
| 2:28 am on Dec 1, 2004 (gmt 0)|
Just a quick question... I notice that many robots.txt files will have something like this:
My question is: What is the point of listing the user-agent when * is already specified? Is this necessary or just being used as a backup?
Thanks in advance!
| 2:53 pm on Dec 1, 2004 (gmt 0)|
|...or just being used as a backup? |
| 3:06 pm on Dec 1, 2004 (gmt 0)|
Trouble is, many people don't bother to read it properly ;)
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved