Just a short note that, if you have mod_rewrite or the like turned on, make sure that you're not *requiring* the trailing slash because Inktomi (don't know about others) will strip it off of the GET request.
Well the problem was that I had mod_rewrite looking for a specific pattern (e.g. /hello/) and Inktomi would request /hello without the trailing slash...so I'd give out a 404 (or actually in my case a 302) to the crawler.
I fixed it in a jiffy so that the rest of the crawl would go Ok, but wanted to give the heads up anyway. It's not a problem for me now.