Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt


panic - 1:23 am on Oct 30, 2003 (gmt 0)


I allow all content on my site to be crawled. With that said, should I simply not use robots.txt, or have one with the following lines :

user-agent: *
allow: *

Would having the latter of the two improve the number of pages crawled by the spiders?

-panic


Thread source:: http://www.webmasterworld.com/robots_txt/182.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com