| Welcome to WebmasterWorld Guest from 188.8.131.52 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
|Subscribe to WebmasterWorld|
| 2:28 am on Dec 1, 2004 (gmt 0)|
Just a quick question... I notice that many robots.txt files will have something like this:
My question is: What is the point of listing the user-agent when * is already specified? Is this necessary or just being used as a backup?
Thanks in advance!
| 2:53 pm on Dec 1, 2004 (gmt 0)|
|...or just being used as a backup? |
| 3:06 pm on Dec 1, 2004 (gmt 0)|
Trouble is, many people don't bother to read it properly ;)
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2013 all rights reserved