Welcome to WebmasterWorld Guest from 107.20.75.63

Forum Moderators: goodroi

Message Too Old, No Replies

Importance of Hotbot

     
4:57 pm on Nov 19, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 30, 2003
posts:170
votes: 0


Hotbot is not (yet at least) paying attention to my disallows with wildcards, which I thought it did. It's going through my site--mostly discussion forum with 100,000 posts--like crazy.

I was thinking about disallowing HotBot completely, but am wondering what the importance of this search engine is. My referrer report for a typical day shows hundreds of people coming form Google, and maybe half google's amount from MSN. I didn't even see Hotbot as a referrer.

What is the importance of Hotbot in the search engine world?

5:05 pm on Nov 19, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Please note that ONLY Google will accept Disallows with wildcards.

You'll need a separate record in robots.txt to control Google vs. the others.

Jim

5:07 pm on Nov 19, 2003 (gmt 0)

Senior Member

joined:July 21, 2000
posts:1402
votes: 0


HotBot?

What HotBot spider is that?

Do you mean Slurp, FAST or Lycos (still seen occassionally)? Or is HotBot spidering itself and if so, with what?

5:32 pm on Nov 19, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 30, 2003
posts:170
votes: 0


It's my forum (Invision) that is telling me that Hotbot is visiting. After reading the above messages, I went into the pHP code and see that they are assigning Inktomi visits to the Hotbot label. I guess HotBot uses Inktomi?

So, I guess I better rephrase my question: How importnat is Inktomi and who do they spider for?

9:08 am on Nov 20, 2003 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:5811
votes: 64


How importnat is Inktomi and who do they spider for?

They are (imo) extremely important. Inktomi results are used by dozens of portals, meta-searches and databases across the internet. They are the back-end choice of MSN, Overture and other big guys. Recently purchased by Yahoo!, Inktomi is almost certain to become an even bigger player in the future.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members