homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

Can't get into Direct Hit or Hotbot
been spidered, but don't appear

 3:38 pm on Jun 2, 2001 (gmt 0)

hey guys,
I'm finally receiving traffic from nearly all search engines, but the two I'm still lacking are hotbot and directhit. My log files indicate that Hotbot has spidered my site 12 times, the first on April 16, and the most recent on June 2 at 1:03 am (today). I've used the DirectHit about 7 or 8 times over the past couple of weeks, but no luck. I thought that DH used ODP data, but maybe I'm just being impatient. Any suggestions are appreciated. Thanks a lot




 3:47 pm on Jun 2, 2001 (gmt 0)

Hey Justin,

I've been scratching my head on the Direct Hit thing myself. I'm getting spidered regularly by the AskJeeves spider but I still can't find myself at DH. Honestly I'm not too worried about DH cuz a good Looksmart listing and good rank in Ink will overcome any click through stuff IMHO.

Perhaps I don't understand DH's purpose well enough but I can't see why they even exist or what effects they have on anything. I just don't see any major, noticeable effects.

As far as HotBot spidering your site you probably see a little guy named "slurp" coming through. He likes money (like the rest of us) and would prefer you register at Position Tech for a better shot at ranking well in INK.


 3:55 pm on Jun 2, 2001 (gmt 0)

thanks for the quick reply, toolman. Yea, that reminds me Ask Jeeves is the other engine I'm not in. Anyway, my main motive for DH is because Lycos showcases their top 5 positions on SERPS (Trying to catch up with the lingo :) ). As far as Position Tech, since it's such a small cost, I'd definitely consider it. I just don't want to harm my top rankings at AOL etc. Thanks


 4:37 pm on Jun 2, 2001 (gmt 0)

Ahhh yes the Lycos thing. It's becoming quite a page of results over there isn't it.

The problem with that little scenario for Lycos is that the click pop for DH is more than likely influenced by MSN more than anything else. So now if you want a good rank in Lycos you gotta get a good listing in Looksmart and pay PT to rank high in INK to get highly clicked in MSN.

Man what a business!


 9:43 pm on Jun 4, 2001 (gmt 0)

Hey guys. Just dropped in on this discussion because we're having a heck of a a time with DH too. Glad to know we're not the only one's out there. I read somewhere that they're feeding a bunch of little guys but it seems that you're not finding the same. So should I not be as worried about it? My hardest one's lately have been AV (which has now listed me), DH and Northern.

By the way do you know what's up with Northern? I don't know why but I can't seem to get in there at all. Any suggestions?


 3:26 am on Jun 5, 2001 (gmt 0)

toolman: I don't see any click-pop links on MSN, just straight hyperlinks (except for the inktomi paid submissions every once in a while - I think that data is just going to Ink).

I was convinced DH data was coming from Hotbot, and I suspect (but I really have no idea) that Lycos could feed the DH clicks, since they seem to be counting also.

jeremy goodrich

 4:04 am on Jun 5, 2001 (gmt 0)

Directhit's cobranded site for MSN [msn.directhit.com]. Hope that helps clear things up.

I read it the other way, too, but after looking and rereading, I think toolman meant the link above.


 12:27 pm on Jun 7, 2001 (gmt 0)

jeremy_goodrich: I am having a hard time understanding this thread. Can you explain a bit more the meaning of your post? I dont see any co-branding or link to MSN from directhit.com.


 12:56 pm on Jun 7, 2001 (gmt 0)

perhaps the URL [msn.directhit.com...]


 2:26 pm on Jun 7, 2001 (gmt 0)

Thanks Steph. I still cant figure out what is going on here. msn.directhit.com does not look anything like search.msn.com and the results are completely different (at least for my keyword). Anybody?


 3:07 pm on Jun 7, 2001 (gmt 0)

i don't know as well, i just don't understand the post of Jeremy_goodrich until i look at the URl and not just the page.
And when i go to their partner's page, i saw this
and i think it could be that

jeremy goodrich

 3:25 pm on Jun 7, 2001 (gmt 0)

I mean that MSN is using the directhit data on their cobranded site, not on the search.msn.com site.

On the search.msn.com site, you see data from looksmart, and inktomi. The only reason sometimes their results look different from other ink partners is that they return more than one result from any given url in the standard search engine results pages.

If you turn this option off, and show only one result per url, the results look more like standard ink. And at the msn.directhit.com site, you get the msn sites by popularity...this is what you get when you click their link "show the most popular sites for (keyword or phrase here.)

Hope this clears up the confusion.


 3:31 pm on Jun 7, 2001 (gmt 0)

yes, this clears up the confusion.


 3:36 pm on Jun 7, 2001 (gmt 0)

Steph/Jeremy: thanks for your helpful posts.


 5:15 pm on Jun 9, 2001 (gmt 0)

I emailed Direct Hit and Hotbot about the problem, and after replying to an automated message from Hotbot, I finally got a person to respond:
"I have added your URL to an internal list which should be indexed within the next week or two." 3 days later, my main page appeared in Hotbot :) . I assume this means my site is now in Inktomi, and it's only a matter of time before its partners show my site? Hopefully, I can get my other pages indexed as well. Thanks



 1:43 am on Jun 10, 2001 (gmt 0)

Justin, Did you ever see the Directhit spider? It is a Mozilla UA, but either has Directhit or Ask Jeeves in the ().


 4:33 pm on Jun 14, 2001 (gmt 0)

I haven't seen the DirectHit spider yet, though I have seen the slurp inktomi spider. Also, when searching for my domain at msn, it appears first in web directory matches with my LS description, but also in web page matches with my meta description. MSN web page matches are fed from MSN, aren't they? So maybe I just have to wait a little bit, and the DH spider will be on its way. Thanks



 7:57 am on Jun 15, 2001 (gmt 0)

Hi Guys
It took me two attempts to get into the DH engine.

First attempt:
* Submit URL
* Check back every now and then to see if in SERPS
* After 2 months finaly appears on page 2 of serps
* Clicked on my url to start the ball rolling on the hit tracker
* Came back again about a month later and my url had gone

Second attempt:
* Submit URL
* check back every day to see if in SERPS
* After 1 month appeard in SERPS
* Clicked on my url to start click tracker
* go back every second or third day and click on URL to keep it active
* Still in SERPS now about 6 months later

1) I get no traffic from DH so im not sure why i keep it up - its the principle i guess

2) In order to keep your url in the SERPS go back every second or third day and click on it. This will keep it alive but will in itself NOT move you up the ranks.

3) DH seems to monitor the IP address of people who click on a SERP and prevents boosting by ignoring repeated clicks from the same ip.

4) To boost your ranking visit with as many different ip addresses as you can and click on your url

5) It will take about a month for your ranking to be recalculated


 12:52 pm on Jun 15, 2001 (gmt 0)

I've also found DH to be a problem, i've come to the conclusion of just giving up, as all my efforts are directed towards getting into excite.

However, is it worth being in DH, as i gather the general concensious is NOT ?

I am interested in big traffic providers for a very specialised field.

What engines or directories does DH feed or contribute too ? (i know about askjeeves)


 2:47 pm on Jun 15, 2001 (gmt 0)

DH provides the top 10 listing for Hotbot and the top 3-5 for Lycos. So, it is actually very important: not in itself, but for what it does for you if you get a top 5 ranking.

jeremy goodrich

 3:51 pm on Jun 15, 2001 (gmt 0)

Great info, thanks for posting that.

Directhit is important for the engines they feed: hotbot, MSN's top 10 popular sites, Lycos popular, Iwon's popular sites, and Ask jeeves.

With all those partners, it is more than worth what it takes to get into the top portion of their results.


 9:34 pm on Jun 25, 2001 (gmt 0)

The Sky Is Falling!!!!!

No not really but to me it is. I have had my site show up in top 10 positions consistently on Fast, HotBot, Lycos, DirectHit (well so so but I was at least ON DirectHit), DMOZ (which I know is a directory so it's unrelated but I might as well mention it) and Mamma. I have now fallen off the face of the planet. I mean literally. I can't find us anywhere. Can anyone tell me why this might have occured? I haven't changed anything on our site so I can't see how this could happen. Is anyone else experiencing anything strange with these engines? Any info would be GREATLY appreciated.



 1:02 pm on Jun 26, 2001 (gmt 0)

nathan_ali - Direct Hit seems to have gone thru a change recently. A site that I had top 5 for over 4 years has dropped off. This would affect Hotbot and Lycos also, as they display the top 5-10 from Direct Hit as their top sites.

Regarding DMOZ, my suspicion is that someone (a competitor) emailed an editor about something spammy about your site. Usually editors are too busy to worry about sites that are already listed unless someone reports them.


 12:10 am on Jun 29, 2001 (gmt 0)

>DH provides the top 10 listing for Hotbot and the top 3-5 for Lycos

This doesn't appear to be the case with either engine, or am I missing something?

Regards, Stavs

jeremy goodrich

 1:02 am on Jun 29, 2001 (gmt 0)

They say "popular sites" on lycos, and they are from Direct Hit. At the bottom of the search engine results page (serp) you'll see the Direct Hit search engine logo.

On Hotbot it is basically the same, they have the top ten listings for those serps that have the direct hit data. They also have the Direct Hit logo on the very bottom of those serps, which also have a logo for a company called "lexiquest" which does some linguistic software.
Hope that helps.


 12:21 pm on Jun 29, 2001 (gmt 0)

stavs - Yesterday they were still out of synch. Hotbot had not updated. Today they are both the same and I am long gone...

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved