| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Subscribe and Support WebmasterWorld|
| 2:28 am on Dec 1, 2004 (gmt 0)|
Just a quick question... I notice that many robots.txt files will have something like this:
My question is: What is the point of listing the user-agent when * is already specified? Is this necessary or just being used as a backup?
Thanks in advance!
| 2:53 pm on Dec 1, 2004 (gmt 0)|
|...or just being used as a backup? |
| 3:06 pm on Dec 1, 2004 (gmt 0)|
Trouble is, many people don't bother to read it properly ;)
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved