Page is a not externally linkable
Tim - 2:52 am on Dec 16, 2003 (gmt 0)
Our FAQ is at: [support.inktomi.com...]
We checked your URL and you do in fact have a
robots.txt at your url. But it does no good at the subdirectory level. It needs to be at the root of (which you have no control over). Your other option is to use meta tags: <META NAME="robots"CONTENT="noindex">
or specifically:http://support.inktomi.com/searchfaq.html#stopslurp
How do I stop your robot / spider (Slurp) from crawling my site?