Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

are the robots.txt examples still valid from WebmasterWorld?

2:59 pm on Jan 28, 2005 (gmt 0)

10+ Year Member

I was wondering if the robotx.txt example provided by WebMasterWorld is still valid to use.

Are the "nice guy/bad guy" listings all set? Is it wise to use this example:

I was also wondering why this is the last portion of the robots.txt:
User-agent: *
Disallow: /

Isn't that saying to block ALL ROBOTS? What about google, msn, yahoo, teoma, and the linking?
Any info will be great.


Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month