- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- robot.txt syntax
Andrew_Thomas - 3:06 pm on Nov 18, 2002 (gmt 0)
If i want my spider to search all my files and directories, is there any benefit from having a robots.txt file. And if so what is the format, as all the examples ive seen only show how to disallow.
Thread source: http://www.webmasterworld.com/robots_txt/126.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com