Welcome to WebmasterWorld Guest from 54.145.136.73

Forum Moderators: Ocean10000 & incrediBILL

WebmasterWorld forum11 Updated and Collated Bot List

   
9:28 am on Oct 4, 2003 (gmt 0)

10+ Year Member



Hi all,
To avoid redundancies of previously treated spiders, I am trying to build a list of all UAs mentioned in this forum11 in this message. I will try to update it regularly. Most of these are collectors, guestbook spammers, copyright bots etc.
Of course it will never be complete. If you think this is a bad idea, I'll stop it. Processed 17 pages so far.
Listed below in alphabetical order.

  • amzn_assoc 2297 [webmasterworld.com]
  • Ano-Kato 2140 [webmasterworld.com]
  • AOLserver ... 2221 [webmasterworld.com], 2131 [webmasterworld.com]
  • arirang_check 2119 [webmasterworld.com]

  • baiduspider 2148 [webmasterworld.com]
  • Batik/1.0 2069 [webmasterworld.com]
  • boitho.com-robot/1.1 2149 [webmasterworld.com]

  • Cityreview Robot 2179 [webmasterworld.com]
  • Checkbot/1.71 2009 [webmasterworld.com]
  • CHIP Explorer HU 2308 [webmasterworld.com]
  • cj.com Spider 2289 [webmasterworld.com]
  • COMBINE 2111 [webmasterworld.com]
  • CopyHunter/1.54 2104 [webmasterworld.com]
  • Crawl_Application 2082 [webmasterworld.com]
  • Custo 2032 [webmasterworld.com]
  • Cxhttp 2051 [webmasterworld.com]

  • Dolly/1.0 2122 [webmasterworld.com]
  • DTS Agent 2305 [webmasterworld.com]

  • EasyDL/... 2189 [webmasterworld.com]
  • EgotoBot/4.8 2269 [webmasterworld.com]
  • Exalead (multipla UA)2203 [webmasterworld.com], 2147 [webmasterworld.com], 2137 [webmasterworld.com]

  • FavOrg 2184 [webmasterworld.com]
  • Fbot/1.1 2267 [webmasterworld.com]
  • Feedster Crawler 2242 [webmasterworld.com]
  • Firefly ... 2059 [webmasterworld.com]
  • Flash Processor 2114 [webmasterworld.com]

  • Gaisbot/3.0 2107 [webmasterworld.com]
  • GalaxyBot 2088 [webmasterworld.com], 2073 [webmasterworld.com]
  • gemima/1.0 2080 [webmasterworld.com]
  • GirafaBot 2161 [webmasterworld.com]
  • GoogleBot (fakes only) 2152 [webmasterworld.com], 2139 [webmasterworld.com], 2120 [webmasterworld.com], 2061 [webmasterworld.com]
  • GornKer Crawler 2075 [webmasterworld.com]

  • http*//www.almaden.ibm.com/cs/crawler 2197 [webmasterworld.com]

  • ia_archiver 2144 [webmasterworld.com]
  • IBSBand 2299 [webmasterworld.com]
  • IE 5.5 Compatible Browser 2030 [webmasterworld.com]
  • Illinois State Tech Labs 2241 [webmasterworld.com]
  • Image Collector V1.0 2292 [webmasterworld.com]
  • Intelliseek 2281 [webmasterworld.com]
  • InternetLinkAgent/3.1 2181 [webmasterworld.com]
  • InternetSeer.com 2278 [webmasterworld.com], 2021 [webmasterworld.com]

  • Jakarta Commons-HttpClient/2.0rc1 2291 [webmasterworld.com]
  • Java/... 2318 [webmasterworld.com], 2295 [webmasterworld.com]

  • Keebler elf 2175 [webmasterworld.com]
  • kuloko-bot 2302 [webmasterworld.com], 2300 [webmasterworld.com]

  • larbin (all kinds of) 2226 [webmasterworld.com]
  • libwww (all kind of) 2160 [webmasterworld.com], 2022 [webmasterworld.com]
  • Linkman 2154 [webmasterworld.com]
  • look.com 2233 [webmasterworld.com]
  • LWP::Simple 2029 [webmasterworld.com]

  • Mac Finder 1.0.38 2048 [webmasterworld.com]
  • MarkWatch/1.0 2035 [webmasterworld.com]
  • Martini 2215 [webmasterworld.com], 2162 [webmasterworld.com], 2166 [webmasterworld.com]
  • MeatEater 1995 [webmasterworld.com]
  • MediaPartners 2112 [webmasterworld.com], 2056 [webmasterworld.com]
  • Mediapartners-Google/2.1 2120 [webmasterworld.com], 2097 [webmasterworld.com]
  • Megite 2259 [webmasterworld.com]
  • MicrosoftPrototypeCrawler 1877 [webmasterworld.com]
  • minibot(NaverRobot)/1.0 2115 [webmasterworld.com], 2152 [webmasterworld.com], 2120 [webmasterworld.com], 2113 [webmasterworld.com]
  • Missouri College Browse 2012 [webmasterworld.com]
  • Mister Pix II 2.10 2220 [webmasterworld.com]
  • IBM_Planetwide 2262 [webmasterworld.com]
  • MnogoSearch 2034 [webmasterworld.com]
  • Mozilla/4.0 (compatible; GoogleToolbar 1.1.60-deleon; Windows 98 SE 4. 2225 [webmasterworld.com]
  • Mozilla/4.0 (compatible; MSIE 5.00; Windows 98 2167 [webmasterworld.com]
  • Mozilla/8.0 2042 [webmasterworld.com]
  • Mozilla 2179 [webmasterworld.com], 2036 [webmasterworld.com]
  • MSIECrawler 2270 [webmasterworld.com], 2109 [webmasterworld.com]
  • Msnbot/0.1 2017 [webmasterworld.com]

  • NameProtect 2236 [webmasterworld.com]
  • NPBot 2130 [webmasterworld.com]
  • NaverRobot -> see minibot
  • net.math.crawler.NetCrawler 2315 [webmasterworld.com]
  • Net Sweeper 2164 [webmasterworld.com]
  • Netscape/PICgrabber 2060 [webmasterworld.com]
  • newskies.net 2158 [webmasterworld.com]
  • NITLE Blog Spider/0.01 1953 [webmasterworld.com]
  • nuSearch 2098 [webmasterworld.com]
  • Nutch... 2301 [webmasterworld.com], 2275 [webmasterworld.com]
  • NY Internet Srvcs 1984 [webmasterworld.com]

  • P.Arthur 1.1 2306 [webmasterworld.com]
  • PersonaPilot/1.00 2324 [webmasterworld.com]
  • PHP/... 2274 [webmasterworld.com]
  • Pita ... 2027 [webmasterworld.com]
  • PlantyNet_WebRobot_V1.9 2245 [webmasterworld.com]
  • Program Shareware 1.0.3 2280 [webmasterworld.com]
  • Python-urllib 2287 [webmasterworld.com], 2057 [webmasterworld.com]

  • QuepasaCreep ... 2204 [webmasterworld.com], 1180 [webmasterworld.com]

  • RPT-HTTPClient/0.3-3 2276 [webmasterworld.com]

  • Searchalot 1980 [webmasterworld.com]
  • SearchSpider.com/1.1 2162 [webmasterworld.com]
  • Sleipnir 2249 [webmasterworld.com]
  • Space Bison/0.02 [fu] (Win67; X; SK) 2319 [webmasterworld.com]
  • SpiderKU/0.9 2170 [webmasterworld.com], 2155 [webmasterworld.com]
  • suchtop-bot-1.14 2235 [webmasterworld.com]
  • Szukacz/... 2081 [webmasterworld.com]

  • Taco Bell 2219 [webmasterworld.com]
  • Teleport Pro 2303 [webmasterworld.com]
  • Terrar-UK_Search robot@terrar.co.uk 2213 [webmasterworld.com]
  • Tide 2310 [webmasterworld.com]
  • toCrawl/UrlDispatcher 2007 [webmasterworld.com]
  • tovero 2013 [webmasterworld.com]

  • UbiCrawler/v0.3beta 2307 [webmasterworld.com]
  • Under the Rainbow ... 2258 [webmasterworld.com], 1989 [webmasterworld.com]
  • Utse/0.04? 2257 [webmasterworld.com]

  • VoilaBOT 2227 [webmasterworld.com]

  • YahooSeeker/1.0 2186 [webmasterworld.com]
  • YellSpider 2248 [webmasterworld.com]

  • Wavepluz 2323 [webmasterworld.com]
  • webbot bot include 2165 [webmasterworld.com]
  • WebHiker/1.0 2182 [webmasterworld.com]
  • Web Link Validator 2003 [webmasterworld.com]
  • WebRACE/1.1 2159 [webmasterworld.com]
  • WebSearchBench 2145 [webmasterworld.com]
  • WebGather 3.0 2046 [webmasterworld.com]
  • WebmasterWorldWebBot 2086 [webmasterworld.com]
  • who am i 2190 [webmasterworld.com]
  • Willow Internet Crawler 2099 [webmasterworld.com]

  • Zao/0.1 1895 [webmasterworld.com]
  • Zealbot 2298 [webmasterworld.com]
  • zeus 41852 webster pro v2.9 win32 2132 [webmasterworld.com]
  • Zibie Spider 0.1 Java/1.4.2 2143 [webmasterworld.com]

    [edited by: Brett_Tabke at 11:55 am (utc) on Oct. 4, 2003]
    [edit reason] fix a couple urls [/edit]

  • 10:37 am on Oct 4, 2003 (gmt 0)

    10+ Year Member



    Oh yes, definitely useful! It would be nice if your list was pinned somehow to the top so it doesn't get buried.
    11:10 am on Oct 4, 2003 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    Nice work, it would be better if it was put in the library of this forum for people to read.
    11:57 am on Oct 4, 2003 (gmt 0)

    10+ Year Member



    great job, bull!
    very usefull.
    12:54 pm on Oct 4, 2003 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    Great list bull :):):)

    There's some User-Agents that have changed name, eg. JetCar -> FlashGet or those bots/technologies/scripts that can have more names, eg. libwww <-> LWP, but in these cases both names should be there. The "Java/..." group is an okay exception, and so is "libwww (all kinds of)" as in these cases, the important part of the UA string is common.

    I see the last entry "Zibie spider" is not listed under "Java/...". It probably should be both places. Anyway, 17 pages down it's already a very good tool :)

    >> If you think this is a bad idea

    No way. It's the best one i've seen for a long time - not many threads make it to my bookmarks :)

    /claus


    Added: There are also some bots and odd UA's that are mentioned in forum 39: [webmasterworld.com...]

    [edited by: claus at 2:37 pm (utc) on Oct. 4, 2003]

    1:26 pm on Oct 4, 2003 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    Good work bull , good to see some WebmasterWorld members spending time on doing some research on WebmasterWorld!

    Sid

    2:54 pm on Oct 4, 2003 (gmt 0)

    WebmasterWorld Administrator brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



    Great post bull, when you press page down more than a few times in these sort of posts you know its good ;)

    Would be even nicer to have IP ranges, companies, first spotting of UA etc etc etc. I'm sure lots of people operate sites with lists/IP's such as this, i'll remember to point them to this thread too :-)

    3:13 pm on Oct 4, 2003 (gmt 0)

    10+ Year Member



    Many thanks for the flowers and constructive suggestions, folks :)

    Being at page 30 now, I wanted to "owner edit" my first post to update it, but it seems impossible. Is it perhaps due to Brett's intervention? Any Admin, please help. Thanks.

    For IP issues like Cyveillance, I'd suggest a separate thread. Yes, I'd like to update it regularly.

    3:19 pm on Oct 4, 2003 (gmt 0)

    10+ Year Member



    Great reference list. Thanks.

    I've also been compiling a master list of bad IPs, Requests, URIs, Referrers, UAs and other such nonsense. It's based on observations here and my own log files. The .htaccess is now a disgusting 35.5kb+ in size, but is pretty comprehensive. If anyone is interested, I can post or stickymail it.

    Mark.

    11:33 pm on Oct 4, 2003 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    Definitively something very much needed, many thanks for the work bull!
    Jens
    9:04 pm on Oct 5, 2003 (gmt 0)

    10+ Year Member



    Hey jan-Bull,

    Great labor intensive work.

    For future reference, after two hours (I think) you can no longer do owner edits. I am not sure when they changed this, but I had a similar problem and that is what was told me.

    I don't think it had anything to do with Brett.

    eTN

    5:15 am on Oct 7, 2003 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    <deleted message by sidyadav>
    1:23 pm on Jul 26, 2004 (gmt 0)

    WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



    and oldie, but a goodie.
    2:40 pm on Jul 26, 2004 (gmt 0)

    10+ Year Member



    Yes, perhaps I should invest some time in an updated one?
    10:31 pm on Jul 26, 2004 (gmt 0)

    10+ Year Member



    Thank you very much, bull.

    zoo

    This 21 message thread spans 2 pages: 21
     

    Featured Threads

    My Threads

    Hot Threads This Week

    Hot Threads This Month