Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

are the robots.txt examples still valid from WebmasterWorld?

2:59 pm on Jan 28, 2005 (gmt 0)

Full Member

10+ Year Member

joined:July 13, 2004
votes: 0

I was wondering if the robotx.txt example provided by WebMasterWorld is still valid to use.

Are the "nice guy/bad guy" listings all set? Is it wise to use this example:

I was also wondering why this is the last portion of the robots.txt:
User-agent: *
Disallow: /

Isn't that saying to block ALL ROBOTS? What about google, msn, yahoo, teoma, and the linking?
Any info will be great.