- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- After using robots.txt none of the bots are crawling my site! Help
- 10:24 am on Mar 1, 2010
so basically you are telling every bot NOT to access pages on your site.
the robots exclusion protocol uses the most specific rule and matches the pattern from left to right.
a blank pattern for Disallow means "disallow NOTHING".
"The Web Robots Pages"
] of the "official" REP site:
To allow all robots complete access
(or just create an empty "/robots.txt" file, or don't use one at all)
you will find similar information on SE help pages regarding robots.txt syntax.
Brought to you by