Importance of Hotbot

[slobizman]

Hotbot is not (yet at least) paying attention to my disallows with wildcards, which I thought it did. It's going through my site--mostly discussion forum with 100,000 posts--like crazy.

I was thinking about disallowing HotBot completely, but am wondering what the importance of this search engine is. My referrer report for a typical day shows hundreds of people coming form Google, and maybe half google's amount from MSN. I didn't even see Hotbot as a referrer.

What is the importance of Hotbot in the search engine world?

[jdMorgan]

Please note that ONLY Google will accept Disallows with wildcards.

You'll need a separate record in robots.txt to control Google vs. the others.

Jim

[makemetop]

HotBot?

What HotBot spider is that?

Do you mean Slurp, FAST or Lycos (still seen occassionally)? Or is HotBot spidering itself and if so, with what?

[slobizman]

It's my forum (Invision) that is telling me that Hotbot is visiting. After reading the above messages, I went into the pHP code and see that they are assigning Inktomi visits to the Hotbot label. I guess HotBot uses Inktomi?

So, I guess I better rephrase my question: How importnat is Inktomi and who do they spider for?

[keyplyr]

How importnat is Inktomi and who do they spider for?

They are (imo) extremely important. Inktomi results are used by dozens of portals, meta-searches and databases across the internet. They are the back-end choice of MSN, Overture and other big guys. Recently purchased by Yahoo!, Inktomi is almost certain to become an even bigger player in the future.