Directhit is crawling hard.

Mozilla/4.06 [en] (Direct Hit Grabber)

[littleman]

I wasn't sure where to put this, so I'll put it here. I don't think the forum moderator should mind.

I see DH coming out of two different class Cs and requesting the last several months of submissions.
This could be nice.

[PeteU]

It is definitely more than just submitted pages, its picking up unsubmitted but outside linked pages as well as traversing the sites internal links.
64.14.65.* new IP range started seeing it this week
I am not clear how they use the data. Would it show up on their site only or other places like Hotbot etc?

[Brett_Tabke]

Anyone feeling generous with an IP range and spider ID for DH?

[littleman]

64.14.65.27 - 30
209.67.252.211 - 216
Both are coming out of Exodus, with no host lookup.
Spider UA -> Mozilla/4.06 [en] (Direct Hit Grabber)
Brett, do you IP deliver to DH?

[Kamikaze]

I've been getting hit pretty hard also. But I am not finding any results from it. I wonder what they're up to.

[Brett_Tabke]

Thanks for the ip's. No, I'm not cloaking (yet) for DH, but going to do a little here and there on some higher ranked hotbot pages.

[redzone]

Kamikaze:
DH usually updates their index around the 20th of each month, or the Monday that falls closest to the 20th...
The heavy spidering seems to be them unclogging the backlog of submissions that have piled up on them.
We had a few thousand URL's crawled over the past week. Submissions were from mid May...

[Kamikaze]

I'm still getting slammed by DG grabber. But I still can't find my site in there. Is anyone able to shake hands well with Direct Hit? Are you guys always finding increase in traffic everytime the grabber comes by?

[PeteU]

Kamikaze, I have noticed some traffic started coming from
msn.directhit.com.
also, had some spidering last week from DirectHit using Ask Jeeves user agent.

[Brett_Tabke]

Littleman wrote in another thread [webmasterworld.com] on Direct Hit
>64.14.65.27 - 30
>209.67.252.211 - 216
>Both are coming out of Exodus, with no host lookup.
>Spider UA -> Mozilla/4.06 [en] (Direct Hit Grabber)

Why do so many search engines spider seem to walk back through Exodus? Who are they?

I've spent the morning going through ip lists and updating our spider ips/agents, and it struck me that there sure were many se's that are going through exodus. Some excite's, lycos, and direct hit's all do. A couple from Fast do to. hmmm.

[NFFC]

"In addition, as a customer of Exodus, FAST will host "The World’s Biggest Search Engine" (http://www.alltheweb.com) at Exodus’ Internet Data Centers in Boston, Mass. and Santa Clara, Calif." [fast.no]