Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- After using robots.txt none of the bots are crawling my site! Help
Staffa - 8:49 pm on Feb 28, 2010 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/4088929.htm
Disallow means not allowing, blocking
* means everybody
so basically you are telling every bot NOT to access pages on your site.
If you want every bot to crawl your site just leave your robots.txt file empty.
Brought to you by WebmasterWorld: http://www.webmasterworld.com