Welcome to WebmasterWorld Guest from 54.227.1.130

Forum Moderators: goodroi

Message Too Old, No Replies

Importance of Hotbot

     

slobizman

4:57 pm on Nov 19, 2003 (gmt 0)

10+ Year Member



Hotbot is not (yet at least) paying attention to my disallows with wildcards, which I thought it did. It's going through my site--mostly discussion forum with 100,000 posts--like crazy.

I was thinking about disallowing HotBot completely, but am wondering what the importance of this search engine is. My referrer report for a typical day shows hundreds of people coming form Google, and maybe half google's amount from MSN. I didn't even see Hotbot as a referrer.

What is the importance of Hotbot in the search engine world?

jdMorgan

5:05 pm on Nov 19, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Please note that ONLY Google will accept Disallows with wildcards.

You'll need a separate record in robots.txt to control Google vs. the others.

Jim

makemetop

5:07 pm on Nov 19, 2003 (gmt 0)



HotBot?

What HotBot spider is that?

Do you mean Slurp, FAST or Lycos (still seen occassionally)? Or is HotBot spidering itself and if so, with what?

slobizman

5:32 pm on Nov 19, 2003 (gmt 0)

10+ Year Member



It's my forum (Invision) that is telling me that Hotbot is visiting. After reading the above messages, I went into the pHP code and see that they are assigning Inktomi visits to the Hotbot label. I guess HotBot uses Inktomi?

So, I guess I better rephrase my question: How importnat is Inktomi and who do they spider for?

keyplyr

9:08 am on Nov 20, 2003 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



How importnat is Inktomi and who do they spider for?

They are (imo) extremely important. Inktomi results are used by dozens of portals, meta-searches and databases across the internet. They are the back-end choice of MSN, Overture and other big guys. Recently purchased by Yahoo!, Inktomi is almost certain to become an even bigger player in the future.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month