Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
Just a short note that, if you have mod_rewrite or the like turned on, make sure that you're not *requiring* the trailing slash because Inktomi (don't know about others) will strip it off of the GET request.
Example: Your link points to [widgets.com...]
Inktomi will request [widgets.com...]
which can screw things up major if you're not careful...
Of course, this is not the proper way to do things--so in this regard Inktomi is broke ;)
I fixed it in a jiffy so that the rest of the crawl would go Ok, but wanted to give the heads up anyway. It's not a problem for me now.