Hotbot is not (yet at least) paying attention to my disallows with wildcards, which I thought it did. It's going through my site--mostly discussion forum with 100,000 posts--like crazy.
I was thinking about disallowing HotBot completely, but am wondering what the importance of this search engine is. My referrer report for a typical day shows hundreds of people coming form Google, and maybe half google's amount from MSN. I didn't even see Hotbot as a referrer.
What is the importance of Hotbot in the search engine world?
It's my forum (Invision) that is telling me that Hotbot is visiting. After reading the above messages, I went into the pHP code and see that they are assigning Inktomi visits to the Hotbot label. I guess HotBot uses Inktomi?
So, I guess I better rephrase my question: How importnat is Inktomi and who do they spider for?
How importnat is Inktomi and who do they spider for?
They are (imo) extremely important. Inktomi results are used by dozens of portals, meta-searches and databases across the internet. They are the back-end choice of MSN, Overture and other big guys. Recently purchased by Yahoo!, Inktomi is almost certain to become an even bigger player in the future.