aakk9999 - 7:54 pm on Sep 9, 2010 (gmt 0)

Google has long maintained their ability to crawl content in directories that are blocked by robots.txt exclusions.

Hi Brett, thanks on replying, do you have any reference to the above? I know they are 'able to', but they should not, should they? Because this is precisely why robots.txt is there, to tell bots "Do not go there please". What I am wondering at reading your reply - are you actually saying that Google has said that they are not honouring robots.txt?

With regards to stopping the access with "brute force" - the site is on IIS6 so no .htaccess (and no ISAPI either).