scooterdude - 8:41 pm on May 14, 2011 (gmt 0)

as far as i know, robots bypass robots.txt when they are following a direct link to a webpage/file.

Some web hosting packages show a linked list of all the files in that hosting package in certain circumstances, this can have interesting results