homepage Welcome to WebmasterWorld Guest from 23.22.59.252
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Internet Archive and Thunderstone? Who are they
Justin

10+ Year Member



 
Msg#: 563 posted 5:03 pm on Apr 16, 2001 (gmt 0)

I just analyzed my log files with NetTracker Pro 4.0, and the following two appeared in my robot/spider section, though I don't know who they are: Thunderstone (IP:63.251.4.43) and Internet Archive (uses several ips: 209.247.40. 105-109 and 165-174. Thanks a lot for any help in advance. Also, does anyone know how I change Googlebot to show up in the robot section and not hosts? Thanks a lot
Justin

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 563 posted 5:09 pm on Apr 16, 2001 (gmt 0)

It sounds like that is the software being using by Metacrawler.
There are a couple of posts in this forum about it.

(sorry justin, we don't allow sig file urls here - drop it on your profile).

btw: welcome to the forums. Nice to see another new user who knows how to read log files drop in.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved