homepage Welcome to WebmasterWorld Guest from 54.242.140.11
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Directhit is crawling hard.
Mozilla/4.06 [en] (Direct Hit Grabber)
littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 95 posted 9:17 pm on Jul 28, 2000 (gmt 0)

I wasn't sure where to put this, so I'll put it here. I don't think the forum moderator should mind.

I see DH coming out of two different class Cs and requesting the last several months of submissions.
This could be nice.


 

PeteU

10+ Year Member



 
Msg#: 95 posted 10:13 pm on Jul 28, 2000 (gmt 0)

It is definitely more than just submitted pages, its picking up unsubmitted but outside linked pages as well as traversing the sites internal links.
64.14.65.* new IP range started seeing it this week
I am not clear how they use the data. Would it show up on their site only or other places like Hotbot etc?



Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 95 posted 12:26 pm on Jul 29, 2000 (gmt 0)

Anyone feeling generous with an IP range and spider ID for DH?

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 95 posted 3:19 pm on Jul 29, 2000 (gmt 0)

64.14.65.27 - 30
209.67.252.211 - 216
Both are coming out of Exodus, with no host lookup.
Spider UA -> Mozilla/4.06 [en] (Direct Hit Grabber)
Brett, do you IP deliver to DH?

Kamikaze

10+ Year Member



 
Msg#: 95 posted 10:44 pm on Aug 1, 2000 (gmt 0)

I've been getting hit pretty hard also. But I am not finding any results from it. I wonder what they're up to.

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 95 posted 11:45 pm on Aug 1, 2000 (gmt 0)

Thanks for the ip's. No, I'm not cloaking (yet) for DH, but going to do a little here and there on some higher ranked hotbot pages.

redzone

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 95 posted 3:46 am on Aug 2, 2000 (gmt 0)

Kamikaze:
DH usually updates their index around the 20th of each month, or the Monday that falls closest to the 20th...
The heavy spidering seems to be them unclogging the backlog of submissions that have piled up on them.
We had a few thousand URL's crawled over the past week. Submissions were from mid May...

Kamikaze

10+ Year Member



 
Msg#: 95 posted 5:47 pm on Aug 24, 2000 (gmt 0)

I'm still getting slammed by DG grabber. But I still can't find my site in there. Is anyone able to shake hands well with Direct Hit? Are you guys always finding increase in traffic everytime the grabber comes by?

PeteU

10+ Year Member



 
Msg#: 95 posted 8:00 pm on Aug 24, 2000 (gmt 0)

Kamikaze, I have noticed some traffic started coming from
msn.directhit.com.
also, had some spidering last week from DirectHit using Ask Jeeves user agent.

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 95 posted 4:51 pm on Sep 6, 2000 (gmt 0)

Littleman wrote in another thread [webmasterworld.com] on Direct Hit
>64.14.65.27 - 30
>209.67.252.211 - 216
>Both are coming out of Exodus, with no host lookup.
>Spider UA -> Mozilla/4.06 [en] (Direct Hit Grabber)

Why do so many search engines spider seem to walk back through Exodus? Who are they?

I've spent the morning going through ip lists and updating our spider ips/agents, and it struck me that there sure were many se's that are going through exodus. Some excite's, lycos, and direct hit's all do. A couple from Fast do to. hmmm.

NFFC

WebmasterWorld Senior Member nffc us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 95 posted 7:25 pm on Sep 6, 2000 (gmt 0)

"In addition, as a customer of Exodus, FAST will host "The World’s Biggest Search Engine" (http://www.alltheweb.com) at Exodus’ Internet Data Centers in Boston, Mass. and Santa Clara, Calif." [fast.no]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved