homepage Welcome to WebmasterWorld Guest from 54.242.200.172
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Internap? Crawling for Inktomi?
Brett_Tabke




msg:397171
 3:43 pm on May 7, 2001 (gmt 0)

Is internap crawling for ink? They are exhibiting crawling behavior:
12.27.166.57
198.107.213.57
206.229.153.57
206.64.105.57
206.98.113.57
207.86.73.57
208.47.242.57
208.51.235.57
216.223.48.225
4.20.90.57

Agent and referer:
[internap.com...]

Partners with Inktomi:
[internap.com...]

 

msgraph




msg:397172
 6:05 pm on May 7, 2001 (gmt 0)

They were hitting a ton of my domains for a few weeks and I gave em a list of my IP's so that they would stop. They weren't eating up much bandwidth, just filling up the raw log files.

I "think" for now they are just scouring the web to find the best route to access various parts of the country or world. Like if there is a lag from point A to B to C to D then re-route the connection so that it goes from point A to E to F to D. This would explain them finding sites by IP address instead of DNS.

They might have some other uses for this in the future if they have not already implemented it. I could imagine someone like Inktomi using this to their advantage on their search engine database.

First they rank all the sites they spidered in terms of relevance, then rank them in terms of connectivity. Whoever has the best of both worlds gets a better listing. Or if a site is down too often then it gets dropped out of the index.

Brett_Tabke




msg:397173
 7:03 pm on May 7, 2001 (gmt 0)

Appears it is for their cache servers thought, and not the se. The se stuff is coming out of Exodus (who I'd love some more info on if anyone has it - who owns them, who started them...etc).

msgraph




msg:397174
 7:29 pm on May 7, 2001 (gmt 0)

>>Appears it is for their cache servers

I agree. From reading through Inktomi's Traffic Server info page it looks like InterNap is a real influence on how their server functions.

Here is some info on them financially if you haven't already come across it.

InterNap [cnnfn.cnn.com]

Mike_Mackin




msg:397175
 7:35 pm on May 7, 2001 (gmt 0)

JANUS CAPITAL CORP may have started them.

[sec.gov...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved