Welcome to WebmasterWorld Guest from 54.204.74.171

Forum Moderators: goodroi

Message Too Old, No Replies

are the robots.txt examples still valid from WebmasterWorld?

   
2:59 pm on Jan 28, 2005 (gmt 0)

10+ Year Member



I was wondering if the robotx.txt example provided by WebMasterWorld is still valid to use.

Are the "nice guy/bad guy" listings all set? Is it wise to use this example:
[searchengineworld.com...]

I was also wondering why this is the last portion of the robots.txt:
User-agent: *
Disallow: /

Isn't that saying to block ALL ROBOTS? What about google, msn, yahoo, teoma, and the linking?
Any info will be great.
Thanks