I am about to ban Fast

[Vishal]

This is what I found on Fast Web site

How can I block your crawler from crawling my site?
You can use the Robots Exclusion Protocol to block the Fast Web Crawler and most other crawlers from visiting your web site.

I am using robots.txt file on one of my web site, where no spiders are allowed, but today, I noticed 24 hits from cr021r01.sac2.fastsearch.net

Am I the only one, or there are others too, who have noticed that these spiders now a days dont' read robots.txt file ?

[fom2001uk]

I don't know about Fast but it's been well documented that Google gets past any robots file. Maybe Fast and others are following suit.

Hopefully somebody will have an answer to this problem.

[Brett_Tabke]

Google and Fast both obey robots.txt as far as I know. Yes, there were occasions in the past where they slipped by robots.txt, however most of those turned out to be the "Delay" factor. They only read robots.txt every so often. Fast has been honoring my robots.txt's that have been in place more than 30days. They are just slow about reading the file.

[mark_roach]

I'll echo what Brett has just said. I have quite recently successfully excluded FAST from a mirror of my site by using robots.txt.