Welcome to WebmasterWorld Guest from 23.22.220.37

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

WebmasterWorld forum11 Updated and Collated Bot List

     
9:28 am on Oct 4, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 3, 2002
posts:566
votes: 0


Hi all,
To avoid redundancies of previously treated spiders, I am trying to build a list of all UAs mentioned in this forum11 in this message. I will try to update it regularly. Most of these are collectors, guestbook spammers, copyright bots etc.
Of course it will never be complete. If you think this is a bad idea, I'll stop it. Processed 17 pages so far.
Listed below in alphabetical order.

  • amzn_assoc 2297 [webmasterworld.com]
  • Ano-Kato 2140 [webmasterworld.com]
  • AOLserver ... 2221 [webmasterworld.com], 2131 [webmasterworld.com]
  • arirang_check 2119 [webmasterworld.com]

  • baiduspider 2148 [webmasterworld.com]
  • Batik/1.0 2069 [webmasterworld.com]
  • boitho.com-robot/1.1 2149 [webmasterworld.com]

  • Cityreview Robot 2179 [webmasterworld.com]
  • Checkbot/1.71 2009 [webmasterworld.com]
  • CHIP Explorer HU 2308 [webmasterworld.com]
  • cj.com Spider 2289 [webmasterworld.com]
  • COMBINE 2111 [webmasterworld.com]
  • CopyHunter/1.54 2104 [webmasterworld.com]
  • Crawl_Application 2082 [webmasterworld.com]
  • Custo 2032 [webmasterworld.com]
  • Cxhttp 2051 [webmasterworld.com]

  • Dolly/1.0 2122 [webmasterworld.com]
  • DTS Agent 2305 [webmasterworld.com]

  • EasyDL/... 2189 [webmasterworld.com]
  • EgotoBot/4.8 2269 [webmasterworld.com]
  • Exalead (multipla UA)2203 [webmasterworld.com], 2147 [webmasterworld.com], 2137 [webmasterworld.com]

  • FavOrg 2184 [webmasterworld.com]
  • Fbot/1.1 2267 [webmasterworld.com]
  • Feedster Crawler 2242 [webmasterworld.com]
  • Firefly ... 2059 [webmasterworld.com]
  • Flash Processor 2114 [webmasterworld.com]

  • Gaisbot/3.0 2107 [webmasterworld.com]
  • GalaxyBot 2088 [webmasterworld.com], 2073 [webmasterworld.com]
  • gemima/1.0 2080 [webmasterworld.com]
  • GirafaBot 2161 [webmasterworld.com]
  • GoogleBot (fakes only) 2152 [webmasterworld.com], 2139 [webmasterworld.com], 2120 [webmasterworld.com], 2061 [webmasterworld.com]
  • GornKer Crawler 2075 [webmasterworld.com]

  • http*//www.almaden.ibm.com/cs/crawler 2197 [webmasterworld.com]

  • ia_archiver 2144 [webmasterworld.com]
  • IBSBand 2299 [webmasterworld.com]
  • IE 5.5 Compatible Browser 2030 [webmasterworld.com]
  • Illinois State Tech Labs 2241 [webmasterworld.com]
  • Image Collector V1.0 2292 [webmasterworld.com]
  • Intelliseek 2281 [webmasterworld.com]
  • InternetLinkAgent/3.1 2181 [webmasterworld.com]
  • InternetSeer.com 2278 [webmasterworld.com], 2021 [webmasterworld.com]

  • Jakarta Commons-HttpClient/2.0rc1 2291 [webmasterworld.com]
  • Java/... 2318 [webmasterworld.com], 2295 [webmasterworld.com]

  • Keebler elf 2175 [webmasterworld.com]
  • kuloko-bot 2302 [webmasterworld.com], 2300 [webmasterworld.com]

  • larbin (all kinds of) 2226 [webmasterworld.com]
  • libwww (all kind of) 2160 [webmasterworld.com], 2022 [webmasterworld.com]
  • Linkman 2154 [webmasterworld.com]
  • look.com 2233 [webmasterworld.com]
  • LWP::Simple 2029 [webmasterworld.com]

  • Mac Finder 1.0.38 2048 [webmasterworld.com]
  • MarkWatch/1.0 2035 [webmasterworld.com]
  • Martini 2215 [webmasterworld.com], 2162 [webmasterworld.com], 2166 [webmasterworld.com]
  • MeatEater 1995 [webmasterworld.com]
  • MediaPartners 2112 [webmasterworld.com], 2056 [webmasterworld.com]
  • Mediapartners-Google/2.1 2120 [webmasterworld.com], 2097 [webmasterworld.com]
  • Megite 2259 [webmasterworld.com]
  • MicrosoftPrototypeCrawler 1877 [webmasterworld.com]
  • minibot(NaverRobot)/1.0 2115 [webmasterworld.com], 2152 [webmasterworld.com], 2120 [webmasterworld.com], 2113 [webmasterworld.com]
  • Missouri College Browse 2012 [webmasterworld.com]
  • Mister Pix II 2.10 2220 [webmasterworld.com]
  • IBM_Planetwide 2262 [webmasterworld.com]
  • MnogoSearch 2034 [webmasterworld.com]
  • Mozilla/4.0 (compatible; GoogleToolbar 1.1.60-deleon; Windows 98 SE 4. 2225 [webmasterworld.com]
  • Mozilla/4.0 (compatible; MSIE 5.00; Windows 98 2167 [webmasterworld.com]
  • Mozilla/8.0 2042 [webmasterworld.com]
  • Mozilla 2179 [webmasterworld.com], 2036 [webmasterworld.com]
  • MSIECrawler 2270 [webmasterworld.com], 2109 [webmasterworld.com]
  • Msnbot/0.1 2017 [webmasterworld.com]

  • NameProtect 2236 [webmasterworld.com]
  • NPBot 2130 [webmasterworld.com]
  • NaverRobot -> see minibot
  • net.math.crawler.NetCrawler 2315 [webmasterworld.com]
  • Net Sweeper 2164 [webmasterworld.com]
  • Netscape/PICgrabber 2060 [webmasterworld.com]
  • newskies.net 2158 [webmasterworld.com]
  • NITLE Blog Spider/0.01 1953 [webmasterworld.com]
  • nuSearch 2098 [webmasterworld.com]
  • Nutch... 2301 [webmasterworld.com], 2275 [webmasterworld.com]
  • NY Internet Srvcs 1984 [webmasterworld.com]

  • P.Arthur 1.1 2306 [webmasterworld.com]
  • PersonaPilot/1.00 2324 [webmasterworld.com]
  • PHP/... 2274 [webmasterworld.com]
  • Pita ... 2027 [webmasterworld.com]
  • PlantyNet_WebRobot_V1.9 2245 [webmasterworld.com]
  • Program Shareware 1.0.3 2280 [webmasterworld.com]
  • Python-urllib 2287 [webmasterworld.com], 2057 [webmasterworld.com]

  • QuepasaCreep ... 2204 [webmasterworld.com], 1180 [webmasterworld.com]

  • RPT-HTTPClient/0.3-3 2276 [webmasterworld.com]

  • Searchalot 1980 [webmasterworld.com]
  • SearchSpider.com/1.1 2162 [webmasterworld.com]
  • Sleipnir 2249 [webmasterworld.com]
  • Space Bison/0.02 [fu] (Win67; X; SK) 2319 [webmasterworld.com]
  • SpiderKU/0.9 2170 [webmasterworld.com], 2155 [webmasterworld.com]
  • suchtop-bot-1.14 2235 [webmasterworld.com]
  • Szukacz/... 2081 [webmasterworld.com]

  • Taco Bell 2219 [webmasterworld.com]
  • Teleport Pro 2303 [webmasterworld.com]
  • Terrar-UK_Search robot@terrar.co.uk 2213 [webmasterworld.com]
  • Tide 2310 [webmasterworld.com]
  • toCrawl/UrlDispatcher 2007 [webmasterworld.com]
  • tovero 2013 [webmasterworld.com]

  • UbiCrawler/v0.3beta 2307 [webmasterworld.com]
  • Under the Rainbow ... 2258 [webmasterworld.com], 1989 [webmasterworld.com]
  • Utse/0.04? 2257 [webmasterworld.com]

  • VoilaBOT 2227 [webmasterworld.com]

  • YahooSeeker/1.0 2186 [webmasterworld.com]
  • YellSpider 2248 [webmasterworld.com]

  • Wavepluz 2323 [webmasterworld.com]
  • webbot bot include 2165 [webmasterworld.com]
  • WebHiker/1.0 2182 [webmasterworld.com]
  • Web Link Validator 2003 [webmasterworld.com]
  • WebRACE/1.1 2159 [webmasterworld.com]
  • WebSearchBench 2145 [webmasterworld.com]
  • WebGather 3.0 2046 [webmasterworld.com]
  • WebmasterWorldWebBot 2086 [webmasterworld.com]
  • who am i 2190 [webmasterworld.com]
  • Willow Internet Crawler 2099 [webmasterworld.com]

  • Zao/0.1 1895 [webmasterworld.com]
  • Zealbot 2298 [webmasterworld.com]
  • zeus 41852 webster pro v2.9 win32 2132 [webmasterworld.com]
  • Zibie Spider 0.1 Java/1.4.2 2143 [webmasterworld.com]

    [edited by: Brett_Tabke at 11:55 am (utc) on Oct. 4, 2003]
    [edit reason] fix a couple urls [/edit]

  • 10:37 am on Oct 4, 2003 (gmt 0)

    Preferred Member

    10+ Year Member

    joined:Aug 11, 2003
    posts:495
    votes: 0


    Oh yes, definitely useful! It would be nice if your list was pinned somehow to the top so it doesn't get buried.
    11:10 am on Oct 4, 2003 (gmt 0)

    Senior Member from ZA 

    WebmasterWorld Senior Member 10+ Year Member

    joined:July 15, 2002
    posts:1720
    votes: 1


    Nice work, it would be better if it was put in the library of this forum for people to read.
    11:57 am on Oct 4, 2003 (gmt 0)

    Junior Member

    10+ Year Member

    joined:Apr 27, 2002
    posts:53
    votes: 0


    great job, bull!
    very usefull.
    12:54 pm on Oct 4, 2003 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member 10+ Year Member

    joined:June 15, 2003
    posts:2395
    votes: 0


    Great list bull :):):)

    There's some User-Agents that have changed name, eg. JetCar -> FlashGet or those bots/technologies/scripts that can have more names, eg. libwww <-> LWP, but in these cases both names should be there. The "Java/..." group is an okay exception, and so is "libwww (all kinds of)" as in these cases, the important part of the UA string is common.

    I see the last entry "Zibie spider" is not listed under "Java/...". It probably should be both places. Anyway, 17 pages down it's already a very good tool :)

    >> If you think this is a bad idea

    No way. It's the best one i've seen for a long time - not many threads make it to my bookmarks :)

    /claus


    Added: There are also some bots and odd UA's that are mentioned in forum 39: [webmasterworld.com...]

    [edited by: claus at 2:37 pm (utc) on Oct. 4, 2003]

    1:26 pm on Oct 4, 2003 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member 10+ Year Member

    joined:July 11, 2003
    posts:955
    votes: 0


    Good work bull , good to see some WebmasterWorld members spending time on doing some research on WebmasterWorld!

    Sid

    2:54 pm on Oct 4, 2003 (gmt 0)

    Moderator from GB 

    WebmasterWorld Administrator brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

    joined:Jan 30, 2002
    posts:4842
    votes: 1


    Great post bull, when you press page down more than a few times in these sort of posts you know its good ;)

    Would be even nicer to have IP ranges, companies, first spotting of UA etc etc etc. I'm sure lots of people operate sites with lists/IP's such as this, i'll remember to point them to this thread too :-)

    3:13 pm on Oct 4, 2003 (gmt 0)

    Preferred Member

    10+ Year Member

    joined:June 3, 2002
    posts:566
    votes: 0


    Many thanks for the flowers and constructive suggestions, folks :)

    Being at page 30 now, I wanted to "owner edit" my first post to update it, but it seems impossible. Is it perhaps due to Brett's intervention? Any Admin, please help. Thanks.

    For IP issues like Cyveillance, I'd suggest a separate thread. Yes, I'd like to update it regularly.

    3:19 pm on Oct 4, 2003 (gmt 0)

    New User

    10+ Year Member

    joined:Jan 19, 2003
    posts:22
    votes: 0


    Great reference list. Thanks.

    I've also been compiling a master list of bad IPs, Requests, URIs, Referrers, UAs and other such nonsense. It's based on observations here and my own log files. The .htaccess is now a disgusting 35.5kb+ in size, but is pretty comprehensive. If anyone is interested, I can post or stickymail it.

    Mark.

    11:33 pm on Oct 4, 2003 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member 10+ Year Member

    joined:Apr 30, 2003
    posts:1067
    votes: 0


    Definitively something very much needed, many thanks for the work bull!
    Jens
    9:04 pm on Oct 5, 2003 (gmt 0)

    Junior Member

    10+ Year Member

    joined:Mar 21, 2003
    posts:116
    votes: 0


    Hey jan-Bull,

    Great labor intensive work.

    For future reference, after two hours (I think) you can no longer do owner edits. I am not sure when they changed this, but I had a similar problem and that is what was told me.

    I don't think it had anything to do with Brett.

    eTN

    5:15 am on Oct 7, 2003 (gmt 0)

    Senior Member

    WebmasterWorld Senior Member 10+ Year Member

    joined:July 11, 2003
    posts:955
    votes: 0


    <deleted message by sidyadav>
    1:23 pm on July 26, 2004 (gmt 0)

    Administrator from US 

    WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

    joined:Sept 21, 1999
    posts:38047
    votes: 11


    and oldie, but a goodie.
    2:40 pm on July 26, 2004 (gmt 0)

    Preferred Member

    10+ Year Member

    joined:June 3, 2002
    posts:566
    votes: 0


    Yes, perhaps I should invest some time in an updated one?
    10:31 pm on July 26, 2004 (gmt 0)

    Junior Member

    10+ Year Member

    joined:Oct 22, 2002
    posts:150
    votes: 0


    Thank you very much, bull.

    zoo

    This 21 message thread spans 2 pages: 21