homepage Welcome to WebmasterWorld Guest from 54.242.231.109
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
WebmasterWorld forum11 Updated and Collated Bot List
bull




msg:404473
 9:28 am on Oct 4, 2003 (gmt 0)

Hi all,
To avoid redundancies of previously treated spiders, I am trying to build a list of all UAs mentioned in this forum11 in this message. I will try to update it regularly. Most of these are collectors, guestbook spammers, copyright bots etc.
Of course it will never be complete. If you think this is a bad idea, I'll stop it. Processed 17 pages so far.
Listed below in alphabetical order.

  • amzn_assoc 2297 [webmasterworld.com]
  • Ano-Kato 2140 [webmasterworld.com]
  • AOLserver ... 2221 [webmasterworld.com], 2131 [webmasterworld.com]
  • arirang_check 2119 [webmasterworld.com]

  • baiduspider 2148 [webmasterworld.com]
  • Batik/1.0 2069 [webmasterworld.com]
  • boitho.com-robot/1.1 2149 [webmasterworld.com]

  • Cityreview Robot 2179 [webmasterworld.com]
  • Checkbot/1.71 2009 [webmasterworld.com]
  • CHIP Explorer HU 2308 [webmasterworld.com]
  • cj.com Spider 2289 [webmasterworld.com]
  • COMBINE 2111 [webmasterworld.com]
  • CopyHunter/1.54 2104 [webmasterworld.com]
  • Crawl_Application 2082 [webmasterworld.com]
  • Custo 2032 [webmasterworld.com]
  • Cxhttp 2051 [webmasterworld.com]

  • Dolly/1.0 2122 [webmasterworld.com]
  • DTS Agent 2305 [webmasterworld.com]

  • EasyDL/... 2189 [webmasterworld.com]
  • EgotoBot/4.8 2269 [webmasterworld.com]
  • Exalead (multipla UA)2203 [webmasterworld.com], 2147 [webmasterworld.com], 2137 [webmasterworld.com]

  • FavOrg 2184 [webmasterworld.com]
  • Fbot/1.1 2267 [webmasterworld.com]
  • Feedster Crawler 2242 [webmasterworld.com]
  • Firefly ... 2059 [webmasterworld.com]
  • Flash Processor 2114 [webmasterworld.com]

  • Gaisbot/3.0 2107 [webmasterworld.com]
  • GalaxyBot 2088 [webmasterworld.com], 2073 [webmasterworld.com]
  • gemima/1.0 2080 [webmasterworld.com]
  • GirafaBot 2161 [webmasterworld.com]
  • GoogleBot (fakes only) 2152 [webmasterworld.com], 2139 [webmasterworld.com], 2120 [webmasterworld.com], 2061 [webmasterworld.com]
  • GornKer Crawler 2075 [webmasterworld.com]

  • http*//www.almaden.ibm.com/cs/crawler 2197 [webmasterworld.com]

  • ia_archiver 2144 [webmasterworld.com]
  • IBSBand 2299 [webmasterworld.com]
  • IE 5.5 Compatible Browser 2030 [webmasterworld.com]
  • Illinois State Tech Labs 2241 [webmasterworld.com]
  • Image Collector V1.0 2292 [webmasterworld.com]
  • Intelliseek 2281 [webmasterworld.com]
  • InternetLinkAgent/3.1 2181 [webmasterworld.com]
  • InternetSeer.com 2278 [webmasterworld.com], 2021 [webmasterworld.com]

  • Jakarta Commons-HttpClient/2.0rc1 2291 [webmasterworld.com]
  • Java/... 2318 [webmasterworld.com], 2295 [webmasterworld.com]

  • Keebler elf 2175 [webmasterworld.com]
  • kuloko-bot 2302 [webmasterworld.com], 2300 [webmasterworld.com]

  • larbin (all kinds of) 2226 [webmasterworld.com]
  • libwww (all kind of) 2160 [webmasterworld.com], 2022 [webmasterworld.com]
  • Linkman 2154 [webmasterworld.com]
  • look.com 2233 [webmasterworld.com]
  • LWP::Simple 2029 [webmasterworld.com]

  • Mac Finder 1.0.38 2048 [webmasterworld.com]
  • MarkWatch/1.0 2035 [webmasterworld.com]
  • Martini 2215 [webmasterworld.com], 2162 [webmasterworld.com], 2166 [webmasterworld.com]
  • MeatEater 1995 [webmasterworld.com]
  • MediaPartners 2112 [webmasterworld.com], 2056 [webmasterworld.com]
  • Mediapartners-Google/2.1 2120 [webmasterworld.com], 2097 [webmasterworld.com]
  • Megite 2259 [webmasterworld.com]
  • MicrosoftPrototypeCrawler 1877 [webmasterworld.com]
  • minibot(NaverRobot)/1.0 2115 [webmasterworld.com], 2152 [webmasterworld.com], 2120 [webmasterworld.com], 2113 [webmasterworld.com]
  • Missouri College Browse 2012 [webmasterworld.com]
  • Mister Pix II 2.10 2220 [webmasterworld.com]
  • IBM_Planetwide 2262 [webmasterworld.com]
  • MnogoSearch 2034 [webmasterworld.com]
  • Mozilla/4.0 (compatible; GoogleToolbar 1.1.60-deleon; Windows 98 SE 4. 2225 [webmasterworld.com]
  • Mozilla/4.0 (compatible; MSIE 5.00; Windows 98 2167 [webmasterworld.com]
  • Mozilla/8.0 2042 [webmasterworld.com]
  • Mozilla 2179 [webmasterworld.com], 2036 [webmasterworld.com]
  • MSIECrawler 2270 [webmasterworld.com], 2109 [webmasterworld.com]
  • Msnbot/0.1 2017 [webmasterworld.com]

  • NameProtect 2236 [webmasterworld.com]
  • NPBot 2130 [webmasterworld.com]
  • NaverRobot -> see minibot
  • net.math.crawler.NetCrawler 2315 [webmasterworld.com]
  • Net Sweeper 2164 [webmasterworld.com]
  • Netscape/PICgrabber 2060 [webmasterworld.com]
  • newskies.net 2158 [webmasterworld.com]
  • NITLE Blog Spider/0.01 1953 [webmasterworld.com]
  • nuSearch 2098 [webmasterworld.com]
  • Nutch... 2301 [webmasterworld.com], 2275 [webmasterworld.com]
  • NY Internet Srvcs 1984 [webmasterworld.com]

  • P.Arthur 1.1 2306 [webmasterworld.com]
  • PersonaPilot/1.00 2324 [webmasterworld.com]
  • PHP/... 2274 [webmasterworld.com]
  • Pita ... 2027 [webmasterworld.com]
  • PlantyNet_WebRobot_V1.9 2245 [webmasterworld.com]
  • Program Shareware 1.0.3 2280 [webmasterworld.com]
  • Python-urllib 2287 [webmasterworld.com], 2057 [webmasterworld.com]

  • QuepasaCreep ... 2204 [webmasterworld.com], 1180 [webmasterworld.com]

  • RPT-HTTPClient/0.3-3 2276 [webmasterworld.com]

  • Searchalot 1980 [webmasterworld.com]
  • SearchSpider.com/1.1 2162 [webmasterworld.com]
  • Sleipnir 2249 [webmasterworld.com]
  • Space Bison/0.02 [fu] (Win67; X; SK) 2319 [webmasterworld.com]
  • SpiderKU/0.9 2170 [webmasterworld.com], 2155 [webmasterworld.com]
  • suchtop-bot-1.14 2235 [webmasterworld.com]
  • Szukacz/... 2081 [webmasterworld.com]

  • Taco Bell 2219 [webmasterworld.com]
  • Teleport Pro 2303 [webmasterworld.com]
  • Terrar-UK_Search robot@terrar.co.uk 2213 [webmasterworld.com]
  • Tide 2310 [webmasterworld.com]
  • toCrawl/UrlDispatcher 2007 [webmasterworld.com]
  • tovero 2013 [webmasterworld.com]

  • UbiCrawler/v0.3beta 2307 [webmasterworld.com]
  • Under the Rainbow ... 2258 [webmasterworld.com], 1989 [webmasterworld.com]
  • Utse/0.04? 2257 [webmasterworld.com]

  • VoilaBOT 2227 [webmasterworld.com]

  • YahooSeeker/1.0 2186 [webmasterworld.com]
  • YellSpider 2248 [webmasterworld.com]

  • Wavepluz 2323 [webmasterworld.com]
  • webbot bot include 2165 [webmasterworld.com]
  • WebHiker/1.0 2182 [webmasterworld.com]
  • Web Link Validator 2003 [webmasterworld.com]
  • WebRACE/1.1 2159 [webmasterworld.com]
  • WebSearchBench 2145 [webmasterworld.com]
  • WebGather 3.0 2046 [webmasterworld.com]
  • WebmasterWorldWebBot 2086 [webmasterworld.com]
  • who am i 2190 [webmasterworld.com]
  • Willow Internet Crawler 2099 [webmasterworld.com]

  • Zao/0.1 1895 [webmasterworld.com]
  • Zealbot 2298 [webmasterworld.com]
  • zeus 41852 webster pro v2.9 win32 2132 [webmasterworld.com]
  • Zibie Spider 0.1 Java/1.4.2 2143 [webmasterworld.com]

    [edited by: Brett_Tabke at 11:55 am (utc) on Oct. 4, 2003]
    [edit reason] fix a couple urls [/edit]

  •  

    BlueSky




    msg:404474
     10:37 am on Oct 4, 2003 (gmt 0)

    Oh yes, definitely useful! It would be nice if your list was pinned somehow to the top so it doesn't get buried.

    creative craig




    msg:404475
     11:10 am on Oct 4, 2003 (gmt 0)

    Nice work, it would be better if it was put in the library of this forum for people to read.

    WebRankInfo




    msg:404476
     11:57 am on Oct 4, 2003 (gmt 0)

    great job, bull!
    very usefull.

    claus




    msg:404477
     12:54 pm on Oct 4, 2003 (gmt 0)

    Great list bull :):):)

    There's some User-Agents that have changed name, eg. JetCar -> FlashGet or those bots/technologies/scripts that can have more names, eg. libwww <-> LWP, but in these cases both names should be there. The "Java/..." group is an okay exception, and so is "libwww (all kinds of)" as in these cases, the important part of the UA string is common.

    I see the last entry "Zibie spider" is not listed under "Java/...". It probably should be both places. Anyway, 17 pages down it's already a very good tool :)

    >> If you think this is a bad idea

    No way. It's the best one i've seen for a long time - not many threads make it to my bookmarks :)

    /claus


    Added: There are also some bots and odd UA's that are mentioned in forum 39: [webmasterworld.com...]

    [edited by: claus at 2:37 pm (utc) on Oct. 4, 2003]

    sidyadav




    msg:404478
     1:26 pm on Oct 4, 2003 (gmt 0)

    Good work bull , good to see some WebmasterWorld members spending time on doing some research on WebmasterWorld!

    Sid

    brotherhood of LAN




    msg:404479
     2:54 pm on Oct 4, 2003 (gmt 0)

    Great post bull, when you press page down more than a few times in these sort of posts you know its good ;)

    Would be even nicer to have IP ranges, companies, first spotting of UA etc etc etc. I'm sure lots of people operate sites with lists/IP's such as this, i'll remember to point them to this thread too :-)

    bull




    msg:404480
     3:13 pm on Oct 4, 2003 (gmt 0)

    Many thanks for the flowers and constructive suggestions, folks :)

    Being at page 30 now, I wanted to "owner edit" my first post to update it, but it seems impossible. Is it perhaps due to Brett's intervention? Any Admin, please help. Thanks.

    For IP issues like Cyveillance, I'd suggest a separate thread. Yes, I'd like to update it regularly.

    FineWare




    msg:404481
     3:19 pm on Oct 4, 2003 (gmt 0)

    Great reference list. Thanks.

    I've also been compiling a master list of bad IPs, Requests, URIs, Referrers, UAs and other such nonsense. It's based on observations here and my own log files. The .htaccess is now a disgusting 35.5kb+ in size, but is pretty comprehensive. If anyone is interested, I can post or stickymail it.

    Mark.

    adfree




    msg:404482
     11:33 pm on Oct 4, 2003 (gmt 0)

    Definitively something very much needed, many thanks for the work bull!
    Jens

    Eric in Tennessee




    msg:404483
     9:04 pm on Oct 5, 2003 (gmt 0)

    Hey jan-Bull,

    Great labor intensive work.

    For future reference, after two hours (I think) you can no longer do owner edits. I am not sure when they changed this, but I had a similar problem and that is what was told me.

    I don't think it had anything to do with Brett.

    eTN

    sidyadav




    msg:404484
     5:15 am on Oct 7, 2003 (gmt 0)

    <deleted message by sidyadav>

    Brett_Tabke




    msg:404485
     1:23 pm on Jul 26, 2004 (gmt 0)

    and oldie, but a goodie.

    bull




    msg:404486
     2:40 pm on Jul 26, 2004 (gmt 0)

    Yes, perhaps I should invest some time in an updated one?

    zooloo




    msg:404487
     10:31 pm on Jul 26, 2004 (gmt 0)

    Thank you very much, bull.

    zoo

    bull




    msg:404488
     4:52 pm on Aug 20, 2004 (gmt 0)

    Time for an update IMHO. 293 UAs total.

  • 8484 Boston Project v 1.0 1836 [webmasterworld.com]
  • AaronCarter/15.0 1680 [webmasterworld.com]
  • AmfibiBOT 1729 [webmasterworld.com]
  • amzn_assoc 2297 [webmasterworld.com]
  • Ano-Kato 2140 [webmasterworld.com]
  • AOLServer 2221 [webmasterworld.com], 2131 [webmasterworld.com], 1789 [webmasterworld.com]
  • arirang_check 2119 [webmasterworld.com]
  • Aruyo/0.01 1786 [webmasterworld.com]
  • AsiaNetBot 1917 [webmasterworld.com]
  • ASPseek/1.2.10 1923 [webmasterworld.com]
  • atSpider 1668 [webmasterworld.com]
  • augurfind 1883 [webmasterworld.com]
  • autoemailspider 1668 [webmasterworld.com]

  • baiduspider 2148 [webmasterworld.com], 1848 [webmasterworld.com]
  • Batik/1.0 2069 [webmasterworld.com]
  • BlackWidow ... 1777 [webmasterworld.com]
  • boitho.com-robot/ ... 2149 [webmasterworld.com], 1951 [webmasterworld.com]

  • Cerberian Drtrs Version-3.1-Build-16 2467 [webmasterworld.com]
  • Checkbot/1.71 2009 [webmasterworld.com]
  • CherryPicker 1668 [webmasterworld.com]
  • CHIP Explorer HU 2308 [webmasterworld.com]
  • Cityreview Robot 2179 [webmasterworld.com]
  • cj.com Spider 2289 [webmasterworld.com], 1799 [webmasterworld.com]
  • ClariaBot/1.0 2495 [webmasterworld.com]
  • Combine/ ... 2111 [webmasterworld.com], 1817 [webmasterworld.com]
  • common::Proxtrans/1.00 f39-2539 [webmasterworld.com]
  • Comodo 1857 [webmasterworld.com]
  • Confuzzledbot/2.0 (+BETA [bot.confuzzled.lu...] 1691 [webmasterworld.com]
  • CopyHunter/... 2104 [webmasterworld.com]
  • Cowbot 0.1 2411 [webmasterworld.com], 2441 [webmasterworld.com], 2438 [webmasterworld.com]
  • Crawl_Application 2082 [webmasterworld.com]
  • Custo 2032 [webmasterworld.com]
  • Cxhttp 2051 [webmasterworld.com]

  • Datum/0.1 1760 [webmasterworld.com]
  • DBrowse 1836 [webmasterworld.com]
  • deepak-USC/ISI f39-2400 [webmasterworld.com]
  • deepak-USC/ISI-1.0 2474 [webmasterworld.com]
  • Demo Bot ... 1836 [webmasterworld.com]
  • Diamond/1.0 2495 [webmasterworld.com]
  • DickBlick 2398 [webmasterworld.com]
  • dLoader(NaverRobot)/1.0 see minibot(NaverRobot)
  • Dolly/1.0 2122 [webmasterworld.com]
  • DSurf15a 1836 [webmasterworld.com]
  • DTS Agent 2305 [webmasterworld.com], 1634 [webmasterworld.com]
  • Dumbot f39-2390 [webmasterworld.com]

  • EasyDL/... 2189 [webmasterworld.com]
  • EasyWebPromotion1.0:+(http*//www.easywebpromotion.com/bot.html) 1658 [webmasterworld.com]
  • EBrowse 1836 [webmasterworld.com]
  • EducateSearch ... 2189 [webmasterworld.com]
  • egothor/3.0a f39-2287 [webmasterworld.com]
  • EgotoBot/4.8 2269 [webmasterworld.com]
  • EliteSys Entry 1668 [webmasterworld.com]
  • Email Spider by AlexW 2403 [webmasterworld.com]
  • ETS v5.1 1927 [webmasterworld.com]
  • Eversion Avenger/37.17 (Chorus/MiX 3.2; 4-bit) 1772 [webmasterworld.com]
  • ExactSeek Crawler 1668 [webmasterworld.com]
  • Exalead ... 2203 [webmasterworld.com], 2147 [webmasterworld.com], 2137 [webmasterworld.com]
  • Exava (exabot@exava.com) 2487 [webmasterworld.com]
  • ExtractorPro 1668 [webmasterworld.com]

  • f00/6.66 [spacy] (HMD; Sol/3; Transhuman OS 2.4i) f39-1440 [webmasterworld.com]
  • Fakezilla f39-2514 [webmasterworld.com]
  • FavOrg 2184 [webmasterworld.com]
  • Fbot/1.1 2267 [webmasterworld.com]
  • FeedBucker 1852 [webmasterworld.com]
  • Feedster Crawler 2242 [webmasterworld.com]
  • Firefly ... 2059 [webmasterworld.com]
  • Flash Processor 2114 [webmasterworld.com]
  • Franklin Locator 1836 [webmasterworld.com]
  • FT Agent 1915 [webmasterworld.com]
  • FunWebProducts f39-2350 [webmasterworld.com]

  • Gaisbot/3.0 2107 [webmasterworld.com]
  • GalaxyBot 2088 [webmasterworld.com], 2073 [webmasterworld.com]
  • gemina/1.0 2080 [webmasterworld.com]
  • Generic 1907 [webmasterworld.com], 1702 [webmasterworld.com]
  • GetRight/4.5e f39-2568 [webmasterworld.com]
  • GoogleBot (fakes only) 2152 [webmasterworld.com], 2139 [webmasterworld.com], 2120 [webmasterworld.com], 2061 [webmasterworld.com], 1824 [webmasterworld.com], 1814 [webmasterworld.com], 1744 [webmasterworld.com]
  • GornKer Crawler 2075 [webmasterworld.com]
  • GrigorBot 0.8 1912 [webmasterworld.com]
  • Gwyncound1-1 1787 [webmasterworld.com]

  • Halo 1963 [webmasterworld.com]
  • HtBrowser 2471 [webmasterworld.com]
  • HTML Works 5.5 1925 [webmasterworld.com]
  • http*//www.almaden.ibm.com/cs/crawler 2197 [webmasterworld.com]
  • http*//www.ctechld.com 1736 [webmasterworld.com]
  • [webmasterworld.com...] 1728 [webmasterworld.com]
  • HTTPLib/1.0 1839 [webmasterworld.com]

  • ia_archiver 2498 [webmasterworld.com]
  • IBM WebExplorer /v0.94 1884 [webmasterworld.com]
  • IBM_Planetwide 2262 [webmasterworld.com]
  • IBSBand 2299 [webmasterworld.com]
  • IBSBand 2299 [webmasterworld.com]
  • IE 5.5 Compatible Browser 2030 [webmasterworld.com]
  • iexplore.exe f39-2422 [webmasterworld.com]
  • Illinois State Tech Labs 2241 [webmasterworld.com]
  • Image Collector V1.0 2292 [webmasterworld.com]
  • Industry Program ... 1828 [webmasterworld.com], 1836 [webmasterworld.com]
  • Infomine Virtual Library Crawler/3.0 (see http*//infomine.ucr.edu/projects/vl_crawler/ f39-1506 [webmasterworld.com]
  • infomine.ucr.edu 2421 [webmasterworld.com]
  • Intelliseek 2281 [webmasterworld.com]
  • Internet Explore 5.x 1668 [webmasterworld.com]
  • InternetLinkAgent/3.1 2181 [webmasterworld.com]
  • InternetSeer.com 2278 [webmasterworld.com], 2021 [webmasterworld.com]
  • Irvine/1.1.1 f39-2413 [webmasterworld.com]
  • IUPU Research Bot 1871 [webmasterworld.com]
  • IUSA Browser 1837 [webmasterworld.com]
  • iVia Site Checker\"/1.0 1506 [webmasterworld.com]

  • Jakarta Commons-HttpClient/2.0rc1 2291 [webmasterworld.com]
  • Jakarta HTTP Client f39-2504 [webmasterworld.com]
  • Java/... 2318 [webmasterworld.com], 2143 [webmasterworld.com], f39-1521 [webmasterworld.com], 1783 [webmasterworld.com], 1869 [webmasterworld.com], 2295 [webmasterworld.com]
  • JetBot/1.0 2510 [webmasterworld.com]

  • K2-Summit 2479 [webmasterworld.com]
  • k2spider 1758 [webmasterworld.com]
  • KaHT 1893 [webmasterworld.com]
  • Kapere 1743 [webmasterworld.com]
  • Keebler elf 2175 [webmasterworld.com]
  • kuloko-bot 2302 [webmasterworld.com], 2300 [webmasterworld.com], 1939 [webmasterworld.com]

  • lachesis ... 1746 [webmasterworld.com]
  • larbin ...(all kinds of) 2226 [webmasterworld.com], 1961 [webmasterworld.com], 1790 [webmasterworld.com]
  • LGE/u8150 f39-2373 [webmasterworld.com]
  • libwww ... (all kind of) f39-2576 [webmasterworld.com], 2160 [webmasterworld.com], 2022 [webmasterworld.com], 1937 [webmasterworld.com], 1885 [webmasterworld.com], 1859 [webmasterworld.com]
  • Lincoln State Web Browser 1836 [webmasterworld.com]
  • Linkman 2154 [webmasterworld.com]
  • LinkSweeper/1.1 1631 [webmasterworld.com]
  • LinkWalker 1668 [webmasterworld.com]
  • LiteBot ... 1764 [webmasterworld.com]
  • look.com 2233 [webmasterworld.com]
  • LookBot 2486 [webmasterworld.com]
  • LWP::Simple 2029 [webmasterworld.com]

  • Mac Finder 1.0.38 2048 [webmasterworld.com], 1818 [webmasterworld.com], 2439 [webmasterworld.com]
  • MacNetwork f39-2305 [webmasterworld.com]
  • Mail Sweeper 1668 [webmasterworld.com]
  • MarkWatch/1.0 2035 [webmasterworld.com], 1825 [webmasterworld.com]
  • Martini 2215 [webmasterworld.com], 2162 [webmasterworld.com]
  • MeatEater 1995 [webmasterworld.com]
  • MediaPartners 2112 [webmasterworld.com], 2056 [webmasterworld.com]
  • Mediapartners-Google/2.1 2110 [webmasterworld.com], 2097 [webmasterworld.com], 1749 [webmasterworld.com]
  • Megite 2259 [webmasterworld.com]
  • Microsoft Data Access Internet Publishing Provider Protocol Discovery 1668 [webmasterworld.com]
  • Microsoft Internet Browser 1930 [webmasterworld.com]
  • Microsoft URL Control - 6.00.8169 1668 [webmasterworld.com], 1698 [webmasterworld.com]
  • Microsoft-WebDAV-MiniRedir/5.1.2600 2460 [webmasterworld.com], f39-2549 [webmasterworld.com]
  • MicrosoftPrototypeCrawler 1877 [webmasterworld.com], 1889 [webmasterworld.com], 1855 [webmasterworld.com]
  • minibot(NaverRobot)/1.0 2115 [webmasterworld.com], 2152 [webmasterworld.com], 2120 [webmasterworld.com], 2113 [webmasterworld.com], 1898 [webmasterworld.com], 1711 [webmasterworld.com]
  • Missauga Locate 1836 [webmasterworld.com]
  • Missigua Locator 1.9 1823 [webmasterworld.com], 1836 [webmasterworld.com]
  • Missouri College Browse 2012 [webmasterworld.com], 1836 [webmasterworld.com]
  • Mister Pix II 2.10 2220 [webmasterworld.com]
  • MnogoSearch 2034 [webmasterworld.com]
  • Moozilla 1680 [webmasterworld.com]
  • Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) 1718 [webmasterworld.com]
  • Mozilla 2179 [webmasterworld.com], 2036 [webmasterworld.com]
  • Mozilla/3.0 (compatible) 1830 [webmasterworld.com], 1763 [webmasterworld.com]
  • Mozilla/3.0 (compatible; Indy Library) 1864 [webmasterworld.com]
  • Mozilla/4.0 (compatible; GoogleToolbar 1.1.60-deleon; Windows 98 SE 4. 2225 [webmasterworld.com]
  • Mozilla/4.0 (compatible; MSIE 5.00; Windows 98 2167 [webmasterworld.com]
  • Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request 1704 [webmasterworld.com]
  • Mozilla/4.0 (compatible; MSIE 7.01; Windows 98) 2480 [webmasterworld.com]
  • Mozilla/4.0 efp@gmx.net 1577 [webmasterworld.com]
  • Mozilla/5.0 (Version: ... Type: ...) 1861 [webmasterworld.com]
  • Mozilla/6.0 (compatible; MSIE 6.0; Windows NT 5.2) 2432 [webmasterworld.com]
  • Mozilla/8 2042 [webmasterworld.com]
  • MSIE 6.0 2354 [webmasterworld.com], 2445 [webmasterworld.com]
  • MSIECrawler 2270 [webmasterworld.com], 2109 [webmasterworld.com]
  • Msnbot/0.1 2017 [webmasterworld.com]
  • MSProxy ... f39-1431 [webmasterworld.com]
  • MSWebPostPostInfoProcessor f39-2447 [webmasterworld.com]
  • munky 1668 [webmasterworld.com]
  • bull




    msg:404489
     5:00 pm on Aug 20, 2004 (gmt 0)


  • NameProtect 2236 [webmasterworld.com]
  • NaverRobot 2471 [webmasterworld.com], -> see minibot
  • NCSA_Beta_1 1808 [webmasterworld.com]
  • Net Sweeper 2164 [webmasterworld.com]
  • net.math.crawler.NetCrawler 2315 [webmasterworld.com]
  • NetNose-Crawler 2.0 1969 [webmasterworld.com], 1845 [webmasterworld.com], 1926 [webmasterworld.com], 1904 [webmasterworld.com], 1688 [webmasterworld.com]
  • Netscape (compatible) f39-2397 [webmasterworld.com]
  • Netscape/PICgrabber 2060 [webmasterworld.com]
  • newskies.net 2158 [webmasterworld.com]
  • NexaBot/1.0 1800 [webmasterworld.com]
  • NG/2.0 f39-2601 [webmasterworld.com]
  • NICErsPRO 1668 [webmasterworld.com]
  • NITLE Blog Spider/0.01 1953 [webmasterworld.com]
  • NPBot 2130 [webmasterworld.com], 1928 [webmasterworld.com], 1633 [webmasterworld.com]
  • NPT 0.0 beta 2461 [webmasterworld.com]
  • nuSearch 2098 [webmasterworld.com]
  • Nutch... 2301 [webmasterworld.com], 2275 [webmasterworld.com], 1667 [webmasterworld.com]
  • Nutscrape/... 1680 [webmasterworld.com]
  • NY Internet Srvcs 1984 [webmasterworld.com]

  • obot 1762 [webmasterworld.com], 1616 [webmasterworld.com]
  • Ocelli/1.0 2417 [webmasterworld.com]
  • openfind ... 1798 [webmasterworld.com]
  • OWR_Crawler 1888 [webmasterworld.com], 1612 [webmasterworld.com]

  • P.Arthur 1.1 2306 [webmasterworld.com]
  • PaperPort GetUrlText f39-1486 [webmasterworld.com]
  • PBrowse 1836 [webmasterworld.com]
  • PersonaPilot/1.00 2324 [webmasterworld.com]
  • PEval 1.4b 1836 [webmasterworld.com]
  • PF Free Web Search Tool 1840 [webmasterworld.com]
  • PHP/... 2274 [webmasterworld.com], 1811 [webmasterworld.com], 1751 [webmasterworld.com]
  • Pita ... 2027 [webmasterworld.com]
  • PlantyNet_WebRobot_V1.9 2245 [webmasterworld.com], 1765 [webmasterworld.com]
  • Plucker/Py-1.4 2473 [webmasterworld.com]
  • Powermarks/3.5 1910 [webmasterworld.com]
  • Production Bot ... 1836 [webmasterworld.com]
  • Program Shareware 1.0.3 [ 2280 [webmasterworld.com], 1924 [webmasterworld.com], 1836 [webmasterworld.com]
  • psbot/... 1757 [webmasterworld.com]
  • PSurf15a 1836 [webmasterworld.com]
  • Python-urllib ... 287 [webmasterworld.com], 2057 [webmasterworld.com], 1571 [webmasterworld.com]

  • Qango.com Web Directory 1936 [webmasterworld.com]
  • QuepasaCreep ... 2204 [webmasterworld.com], 1880 [webmasterworld.com]

  • readwebpage 1726 [webmasterworld.com], 1464 [webmasterworld.com]
  • rico/0.1 1738 [webmasterworld.com]
  • RoboCrawl (www.canadiancontent.net) 1862 [webmasterworld.com]
  • RobotMidareru/0.7libwww-perl/5.65 1859 [webmasterworld.com]
  • Roverbot 1668 [webmasterworld.com]
  • RPT-HTTPClient/0.3-3 2276 [webmasterworld.com]
  • RSurf15a 1836 [webmasterworld.com]
  • Rumours-Agent 1683 [webmasterworld.com]

  • Scooter/3.3Y!CrawlX 2485 [webmasterworld.com]
  • Searchalot 1980 [webmasterworld.com]
  • SearchSpider.com/1.1 2162 [webmasterworld.com]
  • semanticdiscovery/0.1 1732 [webmasterworld.com]
  • SKIZZLE! Distributed Internet Spider v1.0 2502 [webmasterworld.com]
  • Sleipnir 2249 [webmasterworld.com]
  • SpaceBison/0.02 [fu] (Win67; X; SK) 2319 [webmasterworld.com]
  • SpiderKU/0.9 2170 [webmasterworld.com], 2155 [webmasterworld.com]
  • SplatSearch.com 1640 [webmasterworld.com]
  • SSurf15a 1836 [webmasterworld.com]
  • StackRambler 1804 [webmasterworld.com]
  • StripIt 0.2 2430 [webmasterworld.com]
  • suchtop-bot-1.14 2235 [webmasterworld.com]
  • SURF 2490 [webmasterworld.com], f39-2388 [webmasterworld.com]
  • SurveyBot/2.2 1921 [webmasterworld.com]
  • Szukacz/... 2081 [webmasterworld.com]

  • Taco Bell 2219 [webmasterworld.com]
  • TAMU_CS_IRL_CRAWLER/1.0 2496 [webmasterworld.com], 2449 [webmasterworld.com]
  • TECOMAC-Crawler/0.4 1742 [webmasterworld.com]
  • Teleport Pro 2303 [webmasterworld.com]
  • Telesoft 1668 [webmasterworld.com]
  • Terrar-UK_Search robot@terrar.co.uk 2213 [webmasterworld.com]
  • test f39-2528 [webmasterworld.com]
  • TestCrawler/1.0 f39-2385 [webmasterworld.com]
  • Tide ... 2310 [webmasterworld.com], 1919 [webmasterworld.com]
  • timboBot/0.9 1766 [webmasterworld.com]
  • toCrawl/UrlDispatcher 2007 [webmasterworld.com]
  • tovero 2013 [webmasterworld.com]
  • TSW Bot 1.01 f39-2316 [webmasterworld.com]
  • TurnitinBot/1.5 http*//www.turnitin.com/robot/crawlerinfo.html 1752 [webmasterworld.com]

  • UbiCrawler/v0.3beta 2307 [webmasterworld.com]
  • UCmore f39-1457 [webmasterworld.com], 2380 [webmasterworld.com]
  • UdmSearch 3.0.3 1630 [webmasterworld.com]
  • UltraWombat 1803 [webmasterworld.com]
  • Under the Rainbow ... 2258 [webmasterworld.com], 1989 [webmasterworld.com]
  • URL Spider Pro/ ... 1821 [webmasterworld.com]
  • Utse/0.04 2257 [webmasterworld.com]

  • vang.net spider 1.6 (Spider 1.7/site@vang.net) 2437 [webmasterworld.com]
  • VoilaBOT 2227 [webmasterworld.com], 1897 [webmasterworld.com]

  • W3Bot 1.0 2466 [webmasterworld.com]
  • Watchfire WebXM 1.0 1626 [webmasterworld.com]
  • Wavepluz 2323 [webmasterworld.com]
  • WE 8.0 2426 [webmasterworld.com]
  • Web Link Validator 2003 [webmasterworld.com]
  • WebBandit 1668 [webmasterworld.com]
  • webbot bot include 2165 [webmasterworld.com]
  • WebCapture 1793 [webmasterworld.com]
  • WebClippings 1710 [webmasterworld.com]
  • WebCopier ... 1802 [webmasterworld.com]
  • WebcraftBoot 1700 [webmasterworld.com]
  • WebEmailExtrac 1668 [webmasterworld.com]
  • WebFilter Robot 1.0 1805 [webmasterworld.com]
  • WebGather 3.0 2046 [webmasterworld.com]
  • WebGo IS - 2168 f39-1523 [webmasterworld.com]
  • WebHiker/1.0 2182 [webmasterworld.com]
  • WebmasterWorldWebBot 2086 [webmasterworld.com]
  • WebRACE/1.1 2159 [webmasterworld.com]
  • WebSearchBench 2145 [webmasterworld.com]
  • WebStripper 1807 [webmasterworld.com]
  • WEP Search ... 1865 [webmasterworld.com], 1871 [webmasterworld.com], 1836 [webmasterworld.com]
  • who am i 2190 [webmasterworld.com]
  • Willow Internet Crawler 2099 [webmasterworld.com]
  • WIRE/0.1 f39-2297 [webmasterworld.com]
  • www.netfactual.com/survey/ 1846 [webmasterworld.com]
  • Wwwc/1.04 2472 [webmasterworld.com]
  • wwwster/1.2 (Beta, mailto:gue[at]cis.uni-muenchen.de) 2491 [webmasterworld.com]

  • XH p\xa4TC f39-1515 [webmasterworld.com]

  • Yahoo-MMCrawler 2489 [webmasterworld.com], 2464 [webmasterworld.com]
  • YahooSeeker/1.0 2186 [webmasterworld.com]
  • YellCrawl V4.0 f39-2290 [webmasterworld.com]
  • YellSpider 2248 [webmasterworld.com], 1696 [webmasterworld.com]

  • Zao/0.1 1895 [webmasterworld.com]
  • Zealbot 2298 [webmasterworld.com]
  • Zelig/0.4 alpha2 1637 [webmasterworld.com]
  • Zeus 2.6 1756 [webmasterworld.com]
  • zeus 41852 webster pro v2.9 win32 2132 [webmasterworld.com]
  • Zibie Spider 0.1 Java/1.4.2 2143 [webmasterworld.com]

    other related pages on WebMasterWorld:
    The Perfect Ban List [webmasterworld.com]
    Modified "bad-bot" perl script from stapel/jdMorgan/Key_Master [webmasterworld.com]
    How to protect from site copiers like teleport? [webmasterworld.com]
    Does anyone redirect bad bots to scumware sites? [webmasterworld.com]
    robots.txt tutorial [webmasterworld.com]
    UA list collected by member transistor (thanks for this!) : [joseluis.pellicer.org...]

  • Josefu




    msg:404490
     8:21 am on Aug 21, 2004 (gmt 0)

    Um, wow. Thanks Bull : )

    All this gave me an idea - would it be possible to start some sort of 'user contributor' permanent thread (stuck up top) where we all could report bots and 'new' bots and the good/bad actions of each? If the conclusions of each report (final fingering?) could then be referenced added to a Library document much like the result of Bull's hard work - a list of spider names with a link to the details about it - that would make the purpose of this forum very clear and I'm sure all would contribute. A bit more work for the moderator, perhaps...

    Just a thought.

    wilderness




    msg:404491
     1:22 pm on Aug 21, 2004 (gmt 0)

    Jose,
    There are some extensive threads in the archives. The ability to find these depends entirely upon one's knowledge of their existtence or ability to use the site search.

    It wouldn't likely be an effective solution anyway :(
    As much effort as has been taken to take note of Bull's accumulated list some folks still make basic inquiries rather than reading.
    A Good example is a recent five count thread in which two links were provided early on and then later in the thread another participant introduces the same IP range.

    IMO the best thing each of us (who have any time spent here) is to accmulate links which will assist other inquiries rather than extending or promoting new threads.
    As I recall the forum has a date-limit on threads which are no longer active. (Not sure what effect this has on restarting threads which existed in the months the forum was down?)

    Josefu




    msg:404492
     7:33 am on Aug 22, 2004 (gmt 0)

    The ability to find these depends entirely upon one's knowledge of their existtence or ability to use the site search.

    Yup : )

    I know that this place isn't a Webmaster encyclopedia but rather a place to deal with current issues - my wrong. Still, it would be nice to start an always-up-to date 'spider database' somewhere. I'll give it some thought - some of the others I've found are horribly outdated. It would be a great tool in the war on spammers for sure. 'Specially since the web-aware world is doubling every three years - we'll never be able to keep up with all the name-changing and other tactics if we 'laisse aller'. Bots aren't hard to deal with today thanks to Apache - : ) - but fingering them and finding out what they want sometimes is.

    GaryK




    msg:404493
     6:27 pm on Aug 22, 2004 (gmt 0)

    Great list. There are some new ones to me. I currently have a list of nearly 30,000 unique user agents. If it would be helpful to this project please visit the site in my profile and download known-agents.zip. Also, in my browscap.ini file take a look at the Website Strippers category as it contains some naughty bots that aren't listed here. HTH.

    Global Options:
     top home search open messages active posts  
     

    Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved