Page is a not externally linkable
panic - 1:23 am on Oct 30, 2003 (gmt 0)
Would having the latter of the two improve the number of pages crawled by the spiders? -panic
I allow all content on my site to be crawled. With that said, should I simply not use robots.txt, or have one with the following lines : user-agent: *
allow: *