cameleon scraper bot

Different UA for every hit

[Hobbs]

Over a period of 10 days, never more than 7 requests per day, always using a new valid UA for every hit, IP coming from a hosting company.

"Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; .NET CLR 1.0.2914; COM+ 1.0.2204; .NET CLR 1.1.4322)"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; H010818)"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; YComp 5.0.2.5)"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Hotbar 4.3.2.0)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Tucows)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; FREESERVE_IE5)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; BCD2000; FunWebProducts; Hotbar 4.3.5.0)"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Address.com; YPC 3.0.0)"
"Mozilla/4.0 (compatible; MSIE 5.5; MSN 2.5; AOL 8.0; Windows 98)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; H010818)"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; H010818; (R1 1.5))"
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; c_athome; DigExt)"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; YComp 5.0.2.6; FunWebProducts)"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; pro)"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; (R1 1.1))"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; yie6-uk)"
"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT; BT Cellnet)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; Wanadoo 6.1)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; {B0727E40-7BFF-11D8-BF96-00028A14743D})"
"Mozilla/4.0 (compatible; MSIE 6.0; AOL 8.0; Windows NT 5.1; .NET CLR 1.0.3705)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; T312461; 9L5.5; 9LA; .NET CLR 1.0.3705)"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3328; .NET CLR 1.1.4322)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; Agilent Technologies IE5.5 SP2)"
"Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; FREE; Hotbar 4.2.8.0)"

[keyplyr]

Can you divulge just which hosting company and their range?

[Hobbs]

I found it only on one site logs, so I was reluctant to reveal details as it could be targeting only my web site.
but here is the range:

88.191.3.0 - 88.191.248.255

[keyplyr]

Thanks, now I can set a trap and see if it drops in.

[keyplyr]

Now that I look at your example more Hobbs, I saw the same behavior a few days ago from a different range:

[webmasterworld.com...]

I think there are a few dozen download tools that will switch UAs like this. Of course a scraper written in any number of languages can do it as well. Difficult to tell which is which.

[Hobbs]

Yeah keyplyr, could be the same, what gave this one away was the zero image requests, I had blocked the single IP 6 days ago, but it was allowed when I did my routine htaccess flushing and it was allowed back in, today the whole range is permanently blocked by firewall.
Adiu, Salut & A revčire 88.191.

[wilderness]

88.191.3.0 - 88.191.248.255

appears to be nothing more than a colo.
Why not just take out the providers entire range?

[Ocean10000]

Slightly off topic.
I have a theory on where they get the list of User-Agents. They assemble a predefined list of URL's, these URL will be a list of files that contain tons of User-Agents strings from public sites. And they just cycle though all of those User-Agents obtained from those public sites. And they periodically refresh those User-Agents from the same list of predefined of URL containing User-Agents.

On my personal site I have put up a list of known User-Agents which gets downloaded pretty often. But I did notice a surge of downloads from random named User-Agents and known spoofing bots. I finally ended up blocking bots which I could detect so they could not download this file. I have been halfway temped to spike this file with some User-Agents that would be easily traceable by inserting a unique string. And just follow the bred crumbs from there.

[edited by: Ocean10000 at 4:46 am (utc) on April 12, 2008]

[Hobbs]

Ocean:
>tons of User-Agents strings

Here is the UA list I got from that one IP
All starting with the usual:
"Mozilla/4.0 (compatible; MSIE
then:

5.0; AOL 4.0; Windows 98; DigExt)"
5.0; AOL 7.0; Windows 98; DigExt; sureseeker.com)"
5.0; MSNIA; Windows 98; DigExt)"
5.0; Windows 95; c_athome; DigExt)"
5.0; Windows 95; DigExt; AtHome0107)"
5.0; Windows 95; DigExt; sureseeker.com)"
5.0; Windows 98; {3B50E920-F4E7-11D7-8E90-002018525D6C})"
5.0; Windows 98; DigExt; Hotbar 4.1.8.0)"
5.0; Windows 98; DigExt; Hotbar 4.2.13.0)"
5.0; Windows 98; DigExt; Hotbar 4.3.2.0)"
5.0; Windows 98; DigExt; NN5.0.2.4)"
5.0; Windows NT; DigExt; HWT)"
5.01; MSNIA; Windows NT 5.0)"
5.01; Windows 95; 1.22.1 )"
5.01; Windows 95; QXW0332i)"
5.01; Windows 98; BPB32_v2)"
5.01; Windows 98; sureseeker.com)"
5.01; Windows NT 5.0; .NET CLR 1.1.4322)"
5.01; Windows NT 5.0; AIRF)"
5.01; Windows NT 5.0; H010818)"
5.01; Windows NT 5.0; Hotbar 4.2.8.0)"
5.01; Windows NT 5.0; Hotbar 4.4.6.0; .NET CLR 1.1.4322)"
5.01; Windows NT 5.0; Netcom_BA_DD)"
5.01; Windows NT 5.0; T312461)"
5.01; Windows NT 5.0; YComp 5.0.2.5)"
5.01; Windows NT; BT Cellnet)"
5.01; Windows NT; IE501NTAb)"
5.01; Windows NT; JP Morgan)"
5.4; Windows NT 51.8)"
5.5; AOL 7.0; Windows 98; DT)"
5.5; AOL 7.0; Windows 98; DT)"
5.5; CS 2000; Windows 98; .NET CLR 1.1.4322)"
5.5; CS 2000; Windows 98; .NET CLR 1.1.4322)"
5.5; MSN 2.5; AOL 8.0; Windows 98)"
5.5; Windows 95; Hotbar 2.0; TBP_7.0_GS)"
5.5; Windows 95; IE5.x/Winxx/EZN/xx)"
5.5; Windows 95; MSN 6.1; MSNbMSFT; MSNmen-us; MSNc21; v5m)"
5.5; Windows 95; OptusIE55-31)"
5.5; Windows 95; Tucows)"
5.5; Windows 98; {B0727E40-7BFF-11D8-BF96-00028A14743D})"
5.5; Windows 98; Avant Browser [avantbrowser.com]; MyIE2)"
5.5; Windows 98; BCD2000; FunWebProducts; Hotbar 4.3.5.0)"
5.5; Windows 98; Compaq)"
5.5; Windows 98; FREESERVE_IE5)"
5.5; Windows 98; GTCom; YComp 5.0.0.0; H010818)"
5.5; Windows 98; KITV4.7 Wanadoo)"
5.5; Windows 98; PeoplePC 1.0; ISP)"
5.5; Windows 98; QXW03347)"
5.5; Windows 98; QXW0337i)"
5.5; Windows 98; QXW0337t)"
5.5; Windows 98; T312461; DSEBUILD420)"
5.5; Windows 98; Win 9x 4.90; {9D55009A-7E20-4FFB-83D8-F7AC188EC843})"
5.5; Windows 98; Win 9x 4.90; CNE-Internet 02252000)"
5.5; Windows 98; Win 9x 4.90; iPrimus)"
5.5; Windows 98; Win 9x 4.90; KITV4.7 Wanadoo)"
5.5; Windows 98; Win 9x 4.90; MSN 6.1; MSNbBBYZ; MSNmen-us; MSNc11; v5m)"
5.5; Windows 98; Win 9x 4.90; TBP_7.0_GS)"
5.5; Windows 98; Win 9x 4.90; YComp 5.0.2.5)"
5.5; Windows 98; Win 9x 4.90; yie6; Q312461)"
5.5; Windows NT 4.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
5.5; Windows NT 4.0; AT&T WNS5.0; T312461)"
5.5; Windows NT 4.0; Caboto Holding SIM)"
5.5; Windows NT 4.0; H010818)"
5.5; Windows NT 4.0; Hotbar 4.1.8.0)"
5.5; Windows NT 4.0; MyIE2)"
5.5; Windows NT 4.0; Sema UK Telecoms upgrade)"
5.5; Windows NT 4.0; T312461; 9L5.5; 9LA; .NET CLR 1.0.3705)"
5.5; Windows NT 4.0; T312461; 9LA; 9L5.5; .NET CLR 1.0.3705)"
5.5; Windows NT 4.0; T312461; Fireman's Fund Insurance Company"
5.5; Windows NT 4.0; T312461; i-NavFourF)"
5.5; Windows NT 4.0; T312461; YComp 5.0.8.6; .NET CLR 1.1.4322)"
5.5; Windows NT 4.0; TIN ie4a32bit rel.1)"
5.5; Windows NT 5.0) Is that better?"
5.5; Windows NT 5.0; Agilent Technologies IE5.5 SP2)"
5.5; Windows NT 5.0; Agilent Technologies IE5.5 SP2)"
5.5; Windows NT 5.0; BTinternet V8.4; .NET CLR 1.0.3705)"
5.5; Windows NT 5.0; DEUBA)"
5.5; Windows NT 5.0; FREE; Hotbar 4.2.8.0)"
5.5; Windows NT 5.0; H010818; .NET CLR 1.0.3705)"
5.5; Windows NT 5.0; H010818; YComp 5.0.0.0; 1.41.1 )"
5.5; Windows NT 5.0; Hotbar 2.0)"
5.5; Windows NT 5.0; INTERIM)"
5.5; Windows NT 5.0; ITP)"
5.5; Windows NT 5.0; Merrill IE 5.5 SP2 Install Kit; T312461)"
5.5; Windows NT 5.0; QXW0336p)"
5.5; Windows NT 5.0; RA; T312461; .NET CLR 1.1.4322)"
5.5; Windows NT 5.0; swap/1/m9456)"
5.5; Windows NT 5.0; T312461; IE5.5_ SP2_wOLE)"
5.5; Windows NT 5.0; T312461; IE5.5_ SP2_wOLE)"
5.5; Windows NT 5.0; T312461; poing!)"
5.5; Windows NT 5.0; Wanadoo 6.1)"
5.5; Windows NT 5.0; WMITLABS2001-02)"
6.0; AOL 6.0; Windows NT 5.1; Q312461; Circle0701)"
6.0; AOL 8.0; Windows 98; Win 9x 4.90; FunWebProducts)"
6.0; AOL 8.0; Windows NT 5.1; .NET CLR 1.0.3705)"
6.0; AOL 8.0; Windows NT 5.1; CDSource=storecd_01_03; iOpus-I-M)"
6.0; AOL 8.0; Windows NT 5.1; YComp 5.0.2.6; .NET CLR 1.0.3705)"
6.0; AOL 9.0; Windows 98)"
6.0; AOL 9.0; Windows 98; FunWebProducts)"
6.0; AOL 9.0; Windows 98; Hotbar 4.0; .NET CLR 1.0.3705)"
6.0; AOL 9.0; Windows NT 5.0; .NET CLR 1.1.4322)"
6.0; AOL 9.0; Windows NT 5.1; {4E677E10-258C-46CD-A00A-6AC2EF6B5F06})"
6.0; AOL 9.0; Windows NT 5.1; {D2F65954-6A14-43B5-86BE-42556275B763})"
6.0; AOL 9.0; Windows NT 5.1; AT&T CSM6.0; AT&T CSM 6)"
6.0; AOL 9.0; Windows NT 5.1; BellSouth/1.0.0; FunWebProducts)"
6.0; AOL 9.0; Windows NT 5.1; YComp 5.0.2.4; Roadrunner)"
6.0; CS 2000 6.0; Windows NT 5.1)"
6.0; MSN 2.5; Windows NT 5.1; DigExt)"
6.0; MSNIA; Windows 98; Win 9x 4.90; AT&T CSM6.0; .NET CLR 1.0.3705)"
6.0; Windows 98) via NetCache version 3.1.2c-Windows NT"
6.0; Windows 98; .NET CLR 1.0.2914; COM+ 1.0.2204; .NET CLR 1.1.4322)"
6.0; Windows 98; Address.com; YPC 3.0.0)"
6.0; Windows 98; Alexa Toolbar)"
6.0; Windows 98; BezeqNet 5; Zahav 5)"
6.0; Windows 98; CI IE 6.0)"
6.0; Windows 98; Compaq; Roadrunner)"
6.0; Windows 98; DP1000; MSN 6.1; MSNbMSFT; MSNmit-it; MSNc00; v5m)"
6.0; Windows 98; ezn73ie5-1)"
6.0; Windows 98; H010818; (R1 1.5))"
6.0; Windows 98; Hotbar 2.0; Q312461)"
6.0; Windows 98; Hotbar 4.3.2.0; .NET CLR 1.1.4322)"
6.0; Windows 98; IE5.x/Winxx/EZN/xx; YComp 5.0.0.0)"
6.0; Windows 98; MSOCD; AtHome020)"
6.0; Windows 98; MyIE2; .NET CLR 1.0.3705)"
6.0; Windows 98; Optus Internet 1.0.0.7; sureseeker.com; NN5.0.2.4; Hotbar 4.3.1.0)"
6.0; Windows 98; OptusIE5)"
6.0; Windows 98; OptusIE55-27; YComp 5.0.2.5)"
6.0; Windows 98; PeoplePC 1.0; hpford)"
6.0; Windows 98; pro)"
6.0; Windows 98; Q312461; (R1 1.1); .NET CLR 1.0.3705)"
6.0; Windows 98; Q312461; SIK30)"
6.0; Windows 98; Q312461; StarBand Version 1.0)"
6.0; Windows 98; SIK30)"
6.0; Windows 98; Supplied by blueyonder)"
6.0; Windows 98; SYMPA; .NET CLR 1.0.3705)"
6.0; Windows 98; T312461; .NET CLR 1.0.3705)"
6.0; Windows 98; T312461; Hotbar 3.0)"
6.0; Windows 98; TUCOWS)"
6.0; Windows 98; VNIE60)"
6.0; Windows 98; Win 9x 4.90; AT&T WNS5.0 IE5.0.01; AT&T WNS5.0)"
6.0; Windows 98; Win 9x 4.90; BTinternet V8.4; H010818)"
6.0; Windows 98; Win 9x 4.90; CHWIE_NL70)"
6.0; Windows 98; Win 9x 4.90; MSN 6.1; MSNbMSFT; MSNmen-us; MSNc11; MSNc11)"
6.0; Windows 98; Win 9x 4.90; MSOCD; Q312461)"
6.0; Windows 98; Win 9x 4.90; MyIE2; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows 98; Win 9x 4.90; provided by PC How To)"
6.0; Windows 98; Win 9x 4.90; Q312461)"
6.0; Windows 98; Win 9x 4.90; WHCC/0.6)"
6.0; Windows 98; Win 9x 4.90; YComp 5.0.2.6; FunWebProducts)"
6.0; Windows 98; YComp 5.0.2.6)"
6.0; Windows 98; YComp 5.0.8.6; AskBar 2.11; yplus 1.0)"
6.0; Windows 98; YPC 3.0.0)"
6.0; Windows NT 4.0; APC)"
6.0; Windows NT 4.0; PCQuest)"
6.0; Windows NT 5.0) (compatible; MSIE 6.0; Windows NT 5.1; Rogers Hi-Speed Internet; FunWebProducts; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; (R1 1.1); .NET CLR 1.0.3705)"
6.0; Windows NT 5.0; .NET CLR 1.0.3705)"
6.0; Windows NT 5.0; .NET CLR 1.0.3705; .NET CLR 1.0.2914)"
6.0; Windows NT 5.0; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; {2E477D6E-5A11-4310-A08B-1FA3AE0E6C39}; .NET CLR 1.0.2914)"
6.0; Windows NT 5.0; {665CA5DF-6149-4382-9952-7B39C4A56AD1})"
6.0; Windows NT 5.0; {681E4328-7B3E-4531-A841-656E569B041C})"
6.0; Windows NT 5.0; 1&1 Internet AG by USt; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; AskBar 2.11; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; AT&T CSM7.0)"
6.0; Windows NT 5.0; AUDI AG - ASAP)"
6.0; Windows NT 5.0; Avant Browser [avantbrowser.com]; .NET CLR 1.1.4322; .NET CLR 1.0.3705)"
6.0; Windows NT 5.0; brip1; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; BTinternet V8.1; T312461; Q312461)"
6.0; Windows NT 5.0; CDSource=storecd_01_03)"
6.0; Windows NT 5.0; CPChem)"
6.0; Windows NT 5.0; CPChem)"
6.0; Windows NT 5.0; DigExt; Tucows)"
6.0; Windows NT 5.0; FunWebProducts; (R1 1.1); .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; FunWebProducts; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; FunWebProducts-MyWay; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; H010818; YComp 5.0.0.0)"
6.0; Windows NT 5.0; Hewlett-Packard)"
6.0; Windows NT 5.0; Highway 194 Internet Vertriebs GmbH; T312461)"
6.0; Windows NT 5.0; Hotbar 4.1.2.0; .NET CLR 1.0.3705)"
6.0; Windows NT 5.0; MSOCD; AtHomeEN191)"
6.0; Windows NT 5.0; N_O_K_I_A; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; NetCaptor 7.0 Beta 2)"
6.0; Windows NT 5.0; NetCaptor 7.0 Beta 2)"
6.0; Windows NT 5.0; NTHS; .NET CLR 1.0.3705)"
6.0; Windows NT 5.0; PCPLUS; FunWebProducts)"
6.0; Windows NT 5.0; PCUser)"
6.0; Windows NT 5.0; Q312461; (R1 1.1))"
6.0; Windows NT 5.0; Q312461; .NET CLR 1.0.2914)"
6.0; Windows NT 5.0; Q312461; APC)"
6.0; Windows NT 5.0; Q312461; DT; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; Q312461; SWFPAC NSB IE6sp1; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; RTA)"
6.0; Windows NT 5.0; sbcydsl 3.12; YComp 5.0.0.0; FunWebProducts-MyWay)"
6.0; Windows NT 5.0; T312461)"
6.0; Windows NT 5.0; T312461; (R1 1.3); .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; T312461; FunWebProducts-MyWay; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.0; utvi160403)"
6.0; Windows NT 5.0; YComp 5.0.2.4; Supplied by Tesco.net)"
6.0; Windows NT 5.0; yie6-de; YComp 5.0.0.0)"
6.0; Windows NT 5.0; yie6-uk)"
6.0; Windows NT 5.1),gzip(gfe)"
6.0; Windows NT 5.1; .NET CLR 1.0.3328; .NET CLR 1.1.4322)"
6.0; Windows NT 5.1; {55173041-4495-432A-B14A-DC8C4904669C}; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.1; {CSC})"
6.0; Windows NT 5.1; APCMAG)"
6.0; Windows NT 5.1; AskBar 2.11)"
6.0; Windows NT 5.1; AT&T CSM 6; .NET CLR 1.0.3705)"
6.0; Windows NT 5.1; AtHomeEN191)"
6.0; Windows NT 5.1; brip1; .NET CLR 1.0.3705)"
6.0; Windows NT 5.1; CHWIE_NL70)"
6.0; Windows NT 5.1; DigExt; Rogers Hi-Speed Internet)"
6.0; Windows NT 5.1; ESB{10ED3712-AD0D-4BB1-8D3C-0B0E15A671C4}; .NET CLR 1.0.3705)"
6.0; Windows NT 5.1; FGL)"
6.0; Windows NT 5.1; IE 6.05; .NET CLR 1.1.4322)"
6.0; Windows NT 5.1; MyIE2; i-NavFourF; Alexa Toolbar)"
6.0; Windows NT 5.1; OPT-OUT; .NET CLR 1.1.4322)"
6.0; Windows NT 5.1; OptusIE5)"
6.0; Windows NT 5.1; OptusNetCable-02; FunWebProducts; .NET CLR 1.1.4322)"
6.0; Windows NT 5.1; Q312461; COM+ 1.0.2204; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
6.0; Windows NT 5.1; Q312461; Hotbar 4.1.8.0; .NET CLR 1.0.3705)"
6.0; Windows NT 5.1; username; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 2.8)"

[edit: sorted UA's for clarity]

[edited by: Hobbs at 9:31 am (utc) on April 12, 2008]