homepage Welcome to WebmasterWorld Guest from 184.72.82.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Who owns this spider?
joanr




msg:396609
 5:06 pm on Apr 17, 2000 (gmt 0)

I've seen this bot occasionally but the other day it chewed 23 pages or so. Can't find it listed on any of the databases, id please? Thanks.

 

Brett_Tabke




msg:396610
 5:38 pm on Apr 17, 2000 (gmt 0)

And the spiders name/ip address?

joanr




msg:396611
 6:02 pm on Apr 17, 2000 (gmt 0)

Sorry 'bout that, thought I had listed it in the topic description. Anyhow it's called Crawler.de. FunnelWeb lists it as a robot but doesn't give me the ip. Searching through my raw logs shows no results for crawler.de. I can't seem to trace this one. Thanks.

fantomaster




msg:396612
 2:54 am on Apr 19, 2000 (gmt 0)

Crawler.de is a German search engine
(and a pretty aggressive one at that).
You can check it out at:
[crawler.de...]

Here's what we have in our botBase
spider wise:
#--------
#SE Crawler
crawler.de
195.20.224.72
#UA Crawler V 0.2.2 admin@crawler.de
crawlit.crawler.de
195.20.224.73
# -------


Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved