1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 3:21 pm May 4, 2026

Forum Moderators: open

Message Too Old, No Replies

List of recent spiders

Unknown to me

         

Thierry Zoller

11:22 am on Mar 12, 2002 (gmt 0)

10+ Year Member



129.137.233.226 - - [23/Jan/2002:15:08:05 -0500] "GET /robots.txt HTTP/1.0" 404 1431 "-" "larbin_2.2.2 (mjovanov@ececs.uc.edu)"
110.iliad.fr - - [24/Jan/2002:03:27:59 -0500] "GET /robots.txt HTTP/1.1" 404 1418 "-" "Mozilla/3.0"

212.97.42.223 - - [25/Jan/2002:01:30:01 -0500] "GET /robots.txt HTTP/1.0" 404 1423 "-" "Mozilla/4.0 (compatible: FDSE robot)"

spider20.tiscalinet.it - - [25/Jan/2002:09:14:08 -0500] "GET /robots.txt HTTP/1.0" 404 1409 "-" "SearchTone2.0 - IDEARE"

trek2.sv.av.com - - [25/Jan/2002:10:34:06 -0500] "GET /robots.txt HTTP/1.0" 404 1401 "-" "Scooter-W3.1.2"

Altavsita came subsequent 5 days in a row requesting robots.txt getting a 404 then leavin, never crawled-

cygne02.saclay.cea.fr - - [27/Jan/2002:08:40:33 -0500] "GET /robots.txt HTTP/1.0" 404 1447 "-" "larbin_2.6_basileocaml (basile.starynkevitch@cea.fr)"

193.7.255.130 - - [30/Jan/2002:04:03:56 -0500] "GET /robots.txt HTTP/1.1" 404 1451 "-" "KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5)"

galileo.ti.telenor.net - - [01/Feb/2002:19:10:01 -0500] "GET /robots.txt HTTP/1.0" 404 1422 "-" "srch (larbin2.6.0@unspecified.mail)"

213.252.152.12 - - [03/Feb/2002:04:16:09 -0500] "GET /robots.txt HTTP/1.1" 404 1455 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)"

212.113.19.151 - - [04/Feb/2002:01:42:25 -0500] "GET /robots.txt HTTP/1.1" 404 1429 "-" "bumblebee@relevare.com"

cygne02.saclay.cea.fr - - [05/Feb/2002:06:39:25 -0500] "GET /robots.txt HTTP/1.0" 404 1439 "-" "larbin_2.6_basileocaml (basile.starynkevitch@cea.fr)"

robot13.openfind.com - - [08/Feb/2002:21:11:48 -0500] "GET /robots.txt HTTP/1.0" 404 1502 "-" "Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)"

a200042158184.rev.prima.com.ar - - [13/Feb/2002:00:04:42 -0500] "GET /robots.txt HTTP/1.1" 404 1475 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Ciudad Internet; MSIECrawler)"
MSIE Crawler ?

66.28.98.3 - - [13/Feb/2002:00:36:14 -0500] "GET /robots.txt HTTP/1.0" 404 1406 "-" "ia_archiver"

198.139.155.86 - - [13/Feb/2002:09:32:00 -0500] "GET /robots.txt HTTP/1.0" 404 1406 "-" "OliverPerry"

spider20.tiscalinet.it - - [14/Feb/2002:11:42:06 -0500] "GET /robots.txt HTTP/1.0" 404 1417 "-" "SearchTone2.0 - IDEARE"

x1crawler-1-0.x-echo.com - - [14/Feb/2002:13:06:21 -0500] "GET /robots.txt HTTP/1.0" 404 1456 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5"

x1crawler-1-0.x-echo.com - -
[15/Feb/2002:08:55:34 -0500] "GET /robots.txt HTTP/1.0" 404 1456 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5"

cer.yubc.net - - [16/Feb/2002:01:44:43 -0500] "GET /robots.txt HTTP/1.0" 404 1423 "-" "Mozilla/4.0 (compatible: FDSE robot)"

ng1.exalead.com - - [16/Feb/2002:09:30:10 -0500] "GET /robots.txt HTTP/1.0" 404 1393 "-" "NG/1.0"

65-84-203-250.client.dsl.net - - [17/Feb/2002:14:28:17 -0500] "GET /robots.txt HTTP/1.0" 404 1430 "-" "larbin_2.6.0
(larbin2.6.0@unspecified.mail)"

62.114.35.58 - - [18/Feb/2002:08:15:39 -0500] "GET /robots.txt HTTP/1.1" 404 1485 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; WayOut-TeenStuff; Q312461; MSIECrawler)"

[fast.no...]
pixnat06.whizbang.com - - [22/Feb/2002:17:02:44 -0500] "GET /robots.txt HTTP/1.1" 404 1433 "-" "Mozilla/4.7 (compatible; Whizbang)"
65.105.223.11 - - [23/Feb/2002:12:34:12 -0500] "GET /robots.txt HTTP/1.0" 404 1422 "-" "WebFindBot(http://www.web-find.com)"

gr12.nttrd.com - - [01/Mar/2002:20:00:22 -0500] "GET /robots.txt HTTP/1.1" 404 1424 "-" "gazz/2.1 (gazz@nttrd.com)"

ip503c238f.speed.planet.nl - - [04/Mar/2002:02:02:01 -0500] "GET /robots.txt HTTP/1.0" 404 1423 "-" "appie 1.1 (www.walhello.com)"

ng1.exabot.com - - [07/Mar/2002:20:11:43 -0500] "GET /robots.txt HTTP/1.0" 404 1401 "-" "NG/1.0"

acq05.xyleme.com - - [08/Mar/2002:03:25:46 -0500] "GET /robots.txt HTTP/1.1" 404 1436 "-" "cosmos/0.9_(robot@xyleme.com)"

jeremy goodrich

2:44 pm on Mar 12, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to webmasterworld. That's quite a list, and I think some of those have been discussed before.

After doing a site search [searchengineworld.com] you might browse through this forum [webmasterworld.com] a bit, and read some on the previous spider discussions we've had there's a lot here :)

Thanks for the post, and again, welcome to webmasterworld.

volatilegx

6:24 pm on Jun 26, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I've run into the FDSE robot a bit lately and while doing a bit of research, I attempted to visit the IP address/hostname of the bot in my web browser. Occasionally time I tried it (from various IP addresses) I came across the webpage generated by Apache after installation is complete.

I also came across the Hiddenstreet.com search engine. They appear to be using this bot.

I believe it is a distributed spider.

mack

6:36 pm on Jun 26, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I run a small SE and use FDSE as my bot. I set it to adhere to robots.txt

Does anyone have any issues with fdse spidering their sites? i.e. do you find it server friendly or are their problems?

volatilegx

6:41 pm on Jun 26, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I finally found the supplier of the FDSE bot...

http**://www.xav.com/scripts/search/

FDSE is short for Fluid Dynamics Search Engine, which is a distributed search engine script.

I don't have any problems with it... I was just trying to dig up some info on it, in case it was a proprietary bot for a commercial search engine.

mack

7:34 pm on Jun 26, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



FDSE is a fairly easy to impliment search tool, more designed for site search but some sites are using it as a small scale web search???

anyone else using it in here???

brotherhood of LAN

7:42 pm on Jun 26, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



mack, i use it, and if i didnt know any better, the site search in here "is it"

I wouldnt ban FDSE. Many people will use this for a wide range of reasons. You can change the name of the bot etc pretty easily anyway.....

mack

7:51 pm on Jun 26, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Yea I noticed that also about the wmw search feature. The result page had the fdse line "your search foundxxxxx no of results" etc. Well I named my bot Mackster :)

I removed the feature that says "your search for xxx found xxxx no of results" that way no one knows now small my index is :)

mbauser2

9:17 pm on Jun 26, 2002 (gmt 0)

10+ Year Member




robot13.openfind.com - - [08/Feb/2002:21:11:48 -0500] "GET /robots.txt HTTP/1.0" 404 1502 "-" "Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)"

I don't want to sound mean, but you shouldn't really need our help identifying that one, unless you're really, really lazy.

brotherhood of LAN

9:24 pm on Jun 26, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



[webmasterworld.com...]
[webmasterworld.com...]

for those with "larbin" in them

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 3:21 pm May 4, 2026