Page is a not externally linkable
Andrew_Thomas - 3:06 pm on Nov 18, 2002 (gmt 0)
thanx
If i want my spider to search all my files and directories, is there any benefit from having a robots.txt file. And if so what is the format, as all the examples ive seen only show how to disallow.