does she take a look anyway?

[soapystar]

if i ban a directory via robots.txt and then a spider follows a link from another site that goes to that directory...does the spider still look into the folder?

[fathom]

Yes & No (I think)

Example: Googlebot will read your robot.txt once each 24 hours (or upon returning if longer).

If she crawls you first then the directory in the same 24-hours she will likely crawl the page through the link provided but should have residual memory of your text file.

If the directory was first upon entering your site will read the robot.txt first and if instructed leave to page alone.

So she may crawl but will remember your instructions.

My take anyway.

This would be a good question for Googleguy to clarify! ;)

[MHes]

I reckon you can stop a spider indexing, but you can't stop her taking a look....!