mcavic - 4:01 pm on Jul 17, 2006 (gmt 0)
The web is public. Without restricting who can enter your site or what you post on your site, it is all available for anyone, human or robot, to download and process.
This shows where Googlebot found the FTP log. It visited domainname.com/foldername/ and there the Web server gave it a directory listing. Visit that link yourself, and you'll see it. Googlebot then visited all of the links on that page. The?C=D;O=A is a link that re-sorts the directory listing.
Would it work if I have empty index.htm pages, and then disallow them from the robots?
Placing an empty index.htm page will prevent anyone from discovering what files are in the directory. But it won't remove any files that are already in Google's index. But you could create the empty file, then rename the directory and change all of your links.