| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
|are the robots.txt examples still valid from WebmasterWorld?|
| 2:59 pm on Jan 28, 2005 (gmt 0)|
I was wondering if the robotx.txt example provided by WebMasterWorld is still valid to use.
Are the "nice guy/bad guy" listings all set? Is it wise to use this example:
I was also wondering why this is the last portion of the robots.txt:
Isn't that saying to block ALL ROBOTS? What about google, msn, yahoo, teoma, and the linking?
Any info will be great.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved