Welcome to WebmasterWorld Guest from 54.162.105.6

Forum Moderators: goodroi

Message Too Old, No Replies

Importance of Hotbot

     
4:57 pm on Nov 19, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 30, 2003
posts:177
votes: 1


Hotbot is not (yet at least) paying attention to my disallows with wildcards, which I thought it did. It's going through my site--mostly discussion forum with 100,000 posts--like crazy.

I was thinking about disallowing HotBot completely, but am wondering what the importance of this search engine is. My referrer report for a typical day shows hundreds of people coming form Google, and maybe half google's amount from MSN. I didn't even see Hotbot as a referrer.

What is the importance of Hotbot in the search engine world?

5:05 pm on Nov 19, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Please note that ONLY Google will accept Disallows with wildcards.

You'll need a separate record in robots.txt to control Google vs. the others.

Jim

5:07 pm on Nov 19, 2003 (gmt 0)

Senior Member

joined:July 21, 2000
posts:1402
votes: 0


HotBot?

What HotBot spider is that?

Do you mean Slurp, FAST or Lycos (still seen occassionally)? Or is HotBot spidering itself and if so, with what?

5:32 pm on Nov 19, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 30, 2003
posts:177
votes: 1


It's my forum (Invision) that is telling me that Hotbot is visiting. After reading the above messages, I went into the pHP code and see that they are assigning Inktomi visits to the Hotbot label. I guess HotBot uses Inktomi?

So, I guess I better rephrase my question: How importnat is Inktomi and who do they spider for?

9:08 am on Nov 20, 2003 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:11107
votes: 659


How importnat is Inktomi and who do they spider for?

They are (imo) extremely important. Inktomi results are used by dozens of portals, meta-searches and databases across the internet. They are the back-end choice of MSN, Overture and other big guys. Recently purchased by Yahoo!, Inktomi is almost certain to become an even bigger player in the future.