Google & Robots.txt

Basic Robots.txt question

[robdawg]

In your robots .txt file can you specify certain directories that you do not want the spider searching? Also if someone has linked to a page in a directory that you have specified the spider not to see, will the spider check for the robots.txt first or will it follow the link and try to index the page anyway?

Thanks

Rob.

[robdawg]

Any one?

[Clint]

This is covered in many other posts in the robots.txt forum, and you should post this in the correct forum, which is the robots.txt forum. The "My threads" area under our control panel area here isn't working correctly. I posted this info before and recently, but the thread isn't showing up! (what's up with that mods?) It was in the robots.txt forum.

Ok after some searching I finally found it, message #2.
[webmasterworld.com...]

I can't answer your second question.

[dmorison]

Also if someone has linked to a page in a directory that you have specified the spider not to see, will the spider check for the robots.txt first or will it follow the link and try to index the page anyway?

Any self-respecting robot will obey robots.txt regardless of how the link was found. Googlebot certainly does, and i'm sure every other mainstream search engine crawler does too. There would be little point in robots.txt if this were not the case.