Welcome to WebmasterWorld Guest from 184.108.40.206
joined:Jan 12, 2004
Ok after some searching I finally found it, message #2.
I can't answer your second question.
Also if someone has linked to a page in a directory that you have specified the spider not to see, will the spider check for the robots.txt first or will it follow the link and try to index the page anyway?
Any self-respecting robot will obey robots.txt regardless of how the link was found. Googlebot certainly does, and i'm sure every other mainstream search engine crawler does too. There would be little point in robots.txt if this were not the case.