| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
Just a quick question... I notice that many robots.txt files will have something like this:
My question is: What is the point of listing the user-agent when * is already specified? Is this necessary or just being used as a backup?
Thanks in advance!
|...or just being used as a backup? |
Trouble is, many people don't bother to read it properly ;)
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved