Scroll to the bottom...You are informing people to disallow all bots :(
Dreamquick
12:01 pm on Nov 8, 2004 (gmt 0)
Not quite... the rules specified in that file allow a selection of crawlers to index the site (see all the entries above the last line), but if a crawler isn't in that list then it should obey the default rule which is do not crawl.
# This restricts access to only known and registered robots.
