Welcome to WebmasterWorld Guest from 54.159.246.164

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Directhit is crawling hard.

Mozilla/4.06 [en] (Direct Hit Grabber)

   
9:17 pm on Jul 28, 2000 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I wasn't sure where to put this, so I'll put it here. I don't think the forum moderator should mind.

I see DH coming out of two different class Cs and requesting the last several months of submissions.
This could be nice.

10:13 pm on Jul 28, 2000 (gmt 0)

10+ Year Member



It is definitely more than just submitted pages, its picking up unsubmitted but outside linked pages as well as traversing the sites internal links.
64.14.65.* new IP range started seeing it this week
I am not clear how they use the data. Would it show up on their site only or other places like Hotbot etc?


12:26 pm on Jul 29, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Anyone feeling generous with an IP range and spider ID for DH?
3:19 pm on Jul 29, 2000 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



64.14.65.27 - 30
209.67.252.211 - 216
Both are coming out of Exodus, with no host lookup.
Spider UA -> Mozilla/4.06 [en] (Direct Hit Grabber)
Brett, do you IP deliver to DH?
10:44 pm on Aug 1, 2000 (gmt 0)

10+ Year Member



I've been getting hit pretty hard also. But I am not finding any results from it. I wonder what they're up to.
11:45 pm on Aug 1, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Thanks for the ip's. No, I'm not cloaking (yet) for DH, but going to do a little here and there on some higher ranked hotbot pages.
3:46 am on Aug 2, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Kamikaze:
DH usually updates their index around the 20th of each month, or the Monday that falls closest to the 20th...
The heavy spidering seems to be them unclogging the backlog of submissions that have piled up on them.
We had a few thousand URL's crawled over the past week. Submissions were from mid May...
5:47 pm on Aug 24, 2000 (gmt 0)

10+ Year Member



I'm still getting slammed by DG grabber. But I still can't find my site in there. Is anyone able to shake hands well with Direct Hit? Are you guys always finding increase in traffic everytime the grabber comes by?
8:00 pm on Aug 24, 2000 (gmt 0)

10+ Year Member



Kamikaze, I have noticed some traffic started coming from
msn.directhit.com.
also, had some spidering last week from DirectHit using Ask Jeeves user agent.
4:51 pm on Sep 6, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Littleman wrote in another thread [webmasterworld.com] on Direct Hit
>64.14.65.27 - 30
>209.67.252.211 - 216
>Both are coming out of Exodus, with no host lookup.
>Spider UA -> Mozilla/4.06 [en] (Direct Hit Grabber)

Why do so many search engines spider seem to walk back through Exodus? Who are they?

I've spent the morning going through ip lists and updating our spider ips/agents, and it struck me that there sure were many se's that are going through exodus. Some excite's, lycos, and direct hit's all do. A couple from Fast do to. hmmm.