homepage Welcome to WebmasterWorld Guest from 50.19.172.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Importance of Hotbot
slobizman

10+ Year Member



 
Msg#: 201 posted 4:57 pm on Nov 19, 2003 (gmt 0)

Hotbot is not (yet at least) paying attention to my disallows with wildcards, which I thought it did. It's going through my site--mostly discussion forum with 100,000 posts--like crazy.

I was thinking about disallowing HotBot completely, but am wondering what the importance of this search engine is. My referrer report for a typical day shows hundreds of people coming form Google, and maybe half google's amount from MSN. I didn't even see Hotbot as a referrer.

What is the importance of Hotbot in the search engine world?

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 201 posted 5:05 pm on Nov 19, 2003 (gmt 0)

Please note that ONLY Google will accept Disallows with wildcards.

You'll need a separate record in robots.txt to control Google vs. the others.

Jim

makemetop



 
Msg#: 201 posted 5:07 pm on Nov 19, 2003 (gmt 0)

HotBot?

What HotBot spider is that?

Do you mean Slurp, FAST or Lycos (still seen occassionally)? Or is HotBot spidering itself and if so, with what?

slobizman

10+ Year Member



 
Msg#: 201 posted 5:32 pm on Nov 19, 2003 (gmt 0)

It's my forum (Invision) that is telling me that Hotbot is visiting. After reading the above messages, I went into the pHP code and see that they are assigning Inktomi visits to the Hotbot label. I guess HotBot uses Inktomi?

So, I guess I better rephrase my question: How importnat is Inktomi and who do they spider for?

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 201 posted 9:08 am on Nov 20, 2003 (gmt 0)

How importnat is Inktomi and who do they spider for?

They are (imo) extremely important. Inktomi results are used by dozens of portals, meta-searches and databases across the internet. They are the back-end choice of MSN, Overture and other big guys. Recently purchased by Yahoo!, Inktomi is almost certain to become an even bigger player in the future.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved