Hi all,
To avoid redundancies of previously treated spiders, I am trying to build a list of all UAs mentioned in this forum11 in this message. I will try to update it regularly. Most of these are collectors, guestbook spammers, copyright bots etc.
Of course it will never be complete. If you think this is a bad idea, I'll stop it. Processed 17 pages so far.
Listed below in alphabetical order.
amzn_assoc 2297 [webmasterworld.com]
Ano-Kato 2140 [webmasterworld.com]
AOLserver ... 2221 [webmasterworld.com], 2131 [webmasterworld.com]
arirang_check 2119 [webmasterworld.com] baiduspider 2148 [webmasterworld.com]
Batik/1.0 2069 [webmasterworld.com]
boitho.com-robot/1.1 2149 [webmasterworld.com] Cityreview Robot 2179 [webmasterworld.com]
Checkbot/1.71 2009 [webmasterworld.com]
CHIP Explorer HU 2308 [webmasterworld.com]
cj.com Spider 2289 [webmasterworld.com]
COMBINE 2111 [webmasterworld.com]
CopyHunter/1.54 2104 [webmasterworld.com]
Crawl_Application 2082 [webmasterworld.com]
Custo 2032 [webmasterworld.com]
Cxhttp 2051 [webmasterworld.com] Dolly/1.0 2122 [webmasterworld.com]
DTS Agent 2305 [webmasterworld.com] EasyDL/... 2189 [webmasterworld.com]
EgotoBot/4.8 2269 [webmasterworld.com]
Exalead (multipla UA)2203 [webmasterworld.com], 2147 [webmasterworld.com], 2137 [webmasterworld.com] FavOrg 2184 [webmasterworld.com]
Fbot/1.1 2267 [webmasterworld.com]
Feedster Crawler 2242 [webmasterworld.com]
Firefly ... 2059 [webmasterworld.com]
Flash Processor 2114 [webmasterworld.com] Gaisbot/3.0 2107 [webmasterworld.com]
GalaxyBot 2088 [webmasterworld.com], 2073 [webmasterworld.com]
gemima/1.0 2080 [webmasterworld.com]
GirafaBot 2161 [webmasterworld.com]
GoogleBot (fakes only) 2152 [webmasterworld.com], 2139 [webmasterworld.com], 2120 [webmasterworld.com], 2061 [webmasterworld.com]
GornKer Crawler 2075 [webmasterworld.com] http*//www.almaden.ibm.com/cs/crawler 2197 [webmasterworld.com] ia_archiver 2144 [webmasterworld.com]
IBSBand 2299 [webmasterworld.com]
IE 5.5 Compatible Browser 2030 [webmasterworld.com]
Illinois State Tech Labs 2241 [webmasterworld.com]
Image Collector V1.0 2292 [webmasterworld.com]
Intelliseek 2281 [webmasterworld.com]
InternetLinkAgent/3.1 2181 [webmasterworld.com]
InternetSeer.com 2278 [webmasterworld.com], 2021 [webmasterworld.com] Jakarta Commons-HttpClient/2.0rc1 2291 [webmasterworld.com]
Java/... 2318 [webmasterworld.com], 2295 [webmasterworld.com] Keebler elf 2175 [webmasterworld.com]
kuloko-bot 2302 [webmasterworld.com], 2300 [webmasterworld.com] larbin (all kinds of) 2226 [webmasterworld.com]
libwww (all kind of) 2160 [webmasterworld.com], 2022 [webmasterworld.com]
Linkman 2154 [webmasterworld.com]
look.com 2233 [webmasterworld.com]
LWP::Simple 2029 [webmasterworld.com] Mac Finder 1.0.38 2048 [webmasterworld.com]
MarkWatch/1.0 2035 [webmasterworld.com]
Martini 2215 [webmasterworld.com], 2162 [webmasterworld.com], 2166 [webmasterworld.com]
MeatEater 1995 [webmasterworld.com]
MediaPartners 2112 [webmasterworld.com], 2056 [webmasterworld.com]
Mediapartners-Google/2.1 2120 [webmasterworld.com], 2097 [webmasterworld.com]
Megite 2259 [webmasterworld.com]
MicrosoftPrototypeCrawler 1877 [webmasterworld.com]
minibot(NaverRobot)/1.0 2115 [webmasterworld.com], 2152 [webmasterworld.com], 2120 [webmasterworld.com], 2113 [webmasterworld.com]
Missouri College Browse 2012 [webmasterworld.com]
Mister Pix II 2.10 2220 [webmasterworld.com]IBM_Planetwide 2262 [webmasterworld.com]
MnogoSearch 2034 [webmasterworld.com]
Mozilla/4.0 (compatible; GoogleToolbar 1.1.60-deleon; Windows 98 SE 4. 2225 [webmasterworld.com]
Mozilla/4.0 (compatible; MSIE 5.00; Windows 98 2167 [webmasterworld.com]
Mozilla/8.0 2042 [webmasterworld.com]
Mozilla 2179 [webmasterworld.com], 2036 [webmasterworld.com]
MSIECrawler 2270 [webmasterworld.com], 2109 [webmasterworld.com]
Msnbot/0.1 2017 [webmasterworld.com] NameProtect 2236 [webmasterworld.com]
NPBot 2130 [webmasterworld.com]
NaverRobot -> see minibot
net.math.crawler.NetCrawler 2315 [webmasterworld.com]
Net Sweeper 2164 [webmasterworld.com]
Netscape/PICgrabber 2060 [webmasterworld.com]
newskies.net 2158 [webmasterworld.com]
NITLE Blog Spider/0.01 1953 [webmasterworld.com]
nuSearch 2098 [webmasterworld.com]
Nutch... 2301 [webmasterworld.com], 2275 [webmasterworld.com]
NY Internet Srvcs 1984 [webmasterworld.com] P.Arthur 1.1 2306 [webmasterworld.com]
PersonaPilot/1.00 2324 [webmasterworld.com]
PHP/... 2274 [webmasterworld.com]
Pita ... 2027 [webmasterworld.com]
PlantyNet_WebRobot_V1.9 2245 [webmasterworld.com]
Program Shareware 1.0.3 2280 [webmasterworld.com]
Python-urllib 2287 [webmasterworld.com], 2057 [webmasterworld.com] QuepasaCreep ... 2204 [webmasterworld.com], 1180 [webmasterworld.com] RPT-HTTPClient/0.3-3 2276 [webmasterworld.com] Searchalot 1980 [webmasterworld.com]
SearchSpider.com/1.1 2162 [webmasterworld.com]
Sleipnir 2249 [webmasterworld.com]
Space Bison/0.02 [fu] (Win67; X; SK) 2319 [webmasterworld.com]
SpiderKU/0.9 2170 [webmasterworld.com], 2155 [webmasterworld.com]
suchtop-bot-1.14 2235 [webmasterworld.com]
Szukacz/... 2081 [webmasterworld.com] Taco Bell 2219 [webmasterworld.com]
Teleport Pro 2303 [webmasterworld.com]
Terrar-UK_Search robot@terrar.co.uk 2213 [webmasterworld.com]
Tide 2310 [webmasterworld.com]
toCrawl/UrlDispatcher 2007 [webmasterworld.com]
tovero 2013 [webmasterworld.com] UbiCrawler/v0.3beta 2307 [webmasterworld.com]
Under the Rainbow ... 2258 [webmasterworld.com], 1989 [webmasterworld.com]
Utse/0.04? 2257 [webmasterworld.com] VoilaBOT 2227 [webmasterworld.com] YahooSeeker/1.0 2186 [webmasterworld.com]
YellSpider 2248 [webmasterworld.com] Wavepluz 2323 [webmasterworld.com]
webbot bot include 2165 [webmasterworld.com]
WebHiker/1.0 2182 [webmasterworld.com]
Web Link Validator 2003 [webmasterworld.com]
WebRACE/1.1 2159 [webmasterworld.com]
WebSearchBench 2145 [webmasterworld.com]
WebGather 3.0 2046 [webmasterworld.com]
WebmasterWorldWebBot 2086 [webmasterworld.com]
who am i 2190 [webmasterworld.com]
Willow Internet Crawler 2099 [webmasterworld.com] Zao/0.1 1895 [webmasterworld.com]
Zealbot 2298 [webmasterworld.com]
zeus 41852 webster pro v2.9 win32 2132 [webmasterworld.com]
Zibie Spider 0.1 Java/1.4.2 2143 [webmasterworld.com] [edited by: Brett_Tabke at 11:55 am (utc) on Oct. 4, 2003]
[edit reason] fix a couple urls [/edit]
