Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots code
hollyhats - 5:54 pm on Oct 3, 2002 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/140.htm
I read over the link you gave me. Question. What files would I not want to be indexed? So, if you don't add the robot.txt file, isn't it crawling all files?
Brought to you by WebmasterWorld: http://www.webmasterworld.com