homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 92 message thread spans 4 pages: 92 ( [1] 2 3 4 > >     
Server Farms - March 2013
Ongoing WMW server farm report
wilderness




msg:4552057
 10:45 am on Mar 7, 2013 (gmt 0)

Continued from previous thread: [webmasterworld.com...]


The old thread has become too large, and there is no longer any method of linking to individual submissions within threads at Webmaster World, thus making the previous thread useless as a reference (they do come up in the search results).

Joe's Datacenter
JOESDC-02 204.27.56.0 - 204.27.63.255 204.27.56.0/21
JOESDC-01 208.94.240.0 - 208.94.247.255 208.94.240.0/21
JOESDC-01 69.195.128.0 - 69.195.159.255 69.195.128.0/19
JOESDC-01 96.43.128.0 - 96.43.143.255 96.43.128.0/20
JOESDC-01 2604:5800:: - 2604:5800:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

[edited by: incrediBILL at 12:59 am (utc) on Mar 8, 2013]
[edit reason] Added link to previous thread [/edit]

 

wilderness




msg:4552521
 11:11 am on Mar 8, 2013 (gmt 0)

Found some 2009 references in my own records, however nothing here at WW.

Krypt Technologies
VPLSNET 100.43.128.0 - 100.43.191.255 100.43.128.0/18
VPLSNET-EAST 107.6.192.0 - 107.6.255.255 107.6.192.0/18
VPLSNET-EAST 173.214.0.0 - 173.214.127.255 173.214.0.0/17
VPLSNET 174.139.0.0 - 174.139.255.255 174.139.0.0/16
VPLSNET-EAST 184.75.176.0 - 184.75.191.255 184.75.176.0/20
VPLSNET-EAST 184.83.0.0 - 184.83.255.255 184.83.0.0/16
VPLSNET-EAST 184.164.192.0 - 184.164.223.255 184.164.192.0/19
VPLSNET209.11.240.0 - 209.11.255.255 209.11.240.0/20
HER-VPLS-TEMP 65.74.131.0 - 65.74.131.15 65.74.131.0/28

blend27




msg:4552861
 1:12 pm on Mar 9, 2013 (gmt 0)

Agreed! Sifting thru the older thread became a drag...

It would also be totally cool if we had a summary for the previous thread.

Anyway..

88.191.3.0 - 88.191.3.255 FR-DEDIBOX
88.190.45.0 - 88.190.45.255 FR-DEDIBOX

I found several others with in 88.191.3.0 - 88.191.248.255 << the entire range gets nuked on my sites.

These are within 88.160.0.0/11, ProXad ADSL Range

wilderness




msg:4552865
 1:30 pm on Mar 9, 2013 (gmt 0)

totally cool if we had a summary for the previous thread


I'd been saving that job for you ;)

Whilst you there you could do the same for Pfui's Amazon thread.

keyplyr




msg:4552916
 8:23 pm on Mar 9, 2013 (gmt 0)


These are within 88.160.0.0/11, ProXad ADSL Range

Wouldn't want to block French ADSL users, they buy stuff :)

But I've had this range blocked for a few years:

Dedibox, FR
88.190.16.0 - 88.191.131.255
88.190.0.0/15

keyplyr




msg:4552936
 9:41 pm on Mar 9, 2013 (gmt 0)


I also had this one:

Dedibox, FR
88.191.192.0 - 88.191.248.255
88.191.192.0/19
88.191.224.0/20
88.191.240.0/21
88.191.248.0/24

wilderness




msg:4553042
 11:09 am on Mar 10, 2013 (gmt 0)

DirectSpace Networks
DSNETWORKS-001 174.140.160.0 - 174.140.175.255 174.140.160.0/20
DIRECTSPACE 69.163.32.0 - 69.163.47.255 69.163.32.0/20
DIRECTSPACE 2605:EA00:: - 2605:EA00:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

keyplyr




msg:4553219
 11:46 pm on Mar 10, 2013 (gmt 0)

soundcloud.com
178.249.136.0 - 178.249.137.255
178.249.136.0/21

Haven't seen them do an independent crawl (yet) but their users can host audio files, not necessarily their own. The upload hits the owner's server, in this case mine. Don't know who it was attempting to rip my audio files, but their broadband IP address is now blocked, along with Soundcloud. Upon request, Soundcloud removed my property quickly without hassle.

wilderness




msg:4553386
 11:43 am on Mar 11, 2013 (gmt 0)

MULTACOM CORPORATION (one of the ranges is part of a larger SAVVIS range)
Old thread in Google SEO Forum [webmasterworld.com]

MULTA-NET10 100.42.64.0 - 100.42.79.255 100.42.64.0/20
MULTA-NET11 108.166.192.0 - 108.166.223.255 108.166.192.0/19
MULTA-NET14 198.52.96.0 - 198.52.127.255 198.52.96.0/19
MULTA-NET13 198.74.96.0 - 198.74.127.255 198.74.96.0/19
MULTA-NET12 198.148.96.0 - 198.148.127.255 198.148.96.0/19
MULTA-NET1 204.13.152.0 - 204.13.155.255 204.13.152.0/22
MULTA-NET2 204.15.72.0 - 204.15.79.255 204.15.72.0/21
MULTA-NET3 208.64.224.0 - 208.64.231.255 208.64.224.0/21
SAVV-S604440-2 208.162.36.0 - 208.162.39.255 (208.157.192.0 - 208.163.31.255)
MULTA-NET9 216.24.240.0 - 216.24.255.255 216.24.240.0/20
MULTA-NET6 216.127.160.0 - 216.127.191.255 216.127.160.0/19
MULTA-NET4 66.152.160.0 - 66.152.191.255 66.152.160.0/19
MULTA-NET5 72.44.64.0 - 72.44.79.255 72.44.64.0/20
MULTA-NET7 96.43.80.0 - 96.43.95.255 96.43.80.0/20
MULTA-NET8 96.45.160.0 - 96.45.175.255 96.45.160.0/20
MULTA6-BLOCK1 2607:F130:: - 2607:F130:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

vexxhost
RANGE2 199.19.212.0 - 199.19.215.255 199.19.212.0/22
RANGE1 199.204.44.0 - 199.204.47.255 199.204.44.0/22
PV6-RANGE1 2604:E100:: - 2604:E100:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

Enmax Envision Inc. ENMAXENV-BLOCK3 208.98.224.0 - 208.98.255.255
AdFarm ENV-ADF-208-98-254-192 208.98.254.192 - 208.98.254.207

ENMAXENV-BLOCK4 204.12.144.0 - 204.12.159.255 204.12.144.0/20
ENMAXENV-BLOCK3 208.98.224.0 - 208.98.255.255 208.98.224.0/19
ENMAXENV-BLOCK2 72.29.224.0 - 72.29.255.255 72.29.224.0/19
ENMAXENV-IPV6-BLOCK1 2606:B800:: - 2606:B800:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

keyplyr




msg:4553602
 7:55 pm on Mar 11, 2013 (gmt 0)

Don could you explain more about the enmax hits? They're an Alberta, Canada energy utility company for residential and cooperate customers. Odd they would be on your site.

wilderness




msg:4553615
 8:26 pm on Mar 11, 2013 (gmt 0)

Enmax Envision Inc. ENMAXENV-BLOCK3 208.98.224.0 - 208.98.255.255
AdFarm ENV-ADF-208-98-254-192 208.98.254.192 - 208.98.254.207

BTW, Western Canada is a hotbed of historical widgets ;)

dstiles




msg:4553661
 10:33 pm on Mar 11, 2013 (gmt 0)

Soundcloud 178.249.136.0/21 is actually 178.249.136.0 - 178.249.143.255

I had about half the multacom ones but...

204.15.72.0 - I have the whole /16 blocked as various servers.

208.162.36.0 - I have 208.128.0.0 - 208.167.191.255 blocked as trouble.

keyplyr




msg:4553668
 11:03 pm on Mar 11, 2013 (gmt 0)



Soundcloud 178.249.136.0/21 is actually 178.249.136.0 - 178.249.143.255

Ha... I had /21 but for some reason posted /23

keyplyr




msg:4553739
 4:10 am on Mar 12, 2013 (gmt 0)


208.162.36.0 - I have 208.128.0.0 - 208.167.191.255 blocked as trouble.

Savvis sub-lets to various players, some valuable depending on the type of business you're in. I tried blocking their entire range but later changed to a more surgical approach. IMO if you're going to block all of Savvis, might as well block CenturyLink itself.

wilderness




msg:4553975
 5:14 pm on Mar 12, 2013 (gmt 0)

Softcom Technology Consulting
SOFTCOM4UU1 207.176.166.192 - 207.176.166.207 207.176.166.192/28
SOFTCOMTCI 168.144.0.0 - 168.144.255.255 168.144.0.0/16

dstiles




msg:4559632
 5:23 pm on Mar 29, 2013 (gmt 0)

A new OVH range registered a few days ago in Canada...

198.50.128.0 - 198.50.255.255
198.50.128.0/17

keyplyr




msg:4559759
 6:25 am on Mar 30, 2013 (gmt 0)



Netriplex Data Centers/Colo

216.59.0.0 - 216.59.63.255
216.59.0.0/18

wilderness




msg:4559806
 10:13 am on Mar 30, 2013 (gmt 0)

Netriplex Data Centers/Colo


Page 2, submission 4 [webmasterworld.com]

keyplyr




msg:4559885
 7:57 pm on Mar 30, 2013 (gmt 0)

Yup, and they're at it again.

keyplyr




msg:4559977
 7:11 am on Mar 31, 2013 (gmt 0)



Advanced Hosters, UK
46.229.160.0 - 46.229.164.255
46.229.160.0/22
46.229.164.0/24

dstiles




msg:4560066
 7:18 pm on Mar 31, 2013 (gmt 0)

Advanced Hosters actually owns the range 46.229.160.0 - 46.229.175.255, although they are scattered across NL, UA and US...

inetnum: 46.229.160.0 - 46.229.175.255
netname: UA-HALDEX-20110217
descr: Haldex Ltd
country: NL
country: US
country: UA

Haldex's address is an accommodation address at a well-known number in Old Gloucester Street, London - a general address for multiple companies of various repute.

keyplyr




msg:4560465
 3:26 am on Apr 2, 2013 (gmt 0)



Priority Colo
204.11.48.0 - 204.11.55.255
204.11.48.0/21

keyplyr




msg:4560469
 3:52 am on Apr 2, 2013 (gmt 0)



DME Hosting

74.221.208.0 - 74.221.223.255
74.221.208.0/20

dstiles




msg:4561069
 4:28 pm on Apr 3, 2013 (gmt 0)

ColoCrossing

198.46.128.0 - 198.46.255.255
198.46.128.0/17

Registered March this year. Took less than a month to generate an unwanted bot hit. :(

Andem




msg:4561911
 1:46 am on Apr 6, 2013 (gmt 0)

Thanks to everybody for their input.

After some really nasty spam from OVH and months of seeing Hetzner IPs (your-server.de) in my logs, I decided to finally block them. I couldn't find much except for some snippets from this thread and its predecessor, so I did some digging.

I've formatted my list for my nginx conf:


# OVH
deny 5.39.0.0/17;
deny 5.135.0.0/16;
deny 8.7.244.0/24;
deny 8.18.128.0/24;
deny 8.18.136.0/21;
deny 8.18.172.0/24;
deny 8.20.110.0/24;
deny 8.21.41.0/24;
deny 8.24.8.0/21;
deny 8.26.94.0/24;
deny 8.29.224.0/24;
deny 8.30.208.0/21;
deny 8.33.96.0/21;
deny 8.33.128.0/21;
deny 8.33.136.0/24;
deny 8.33.137.0/24;
deny 37.49.226.0/24;
deny 37.49.227.0/24;
deny 37.59.0.0/16;
deny 37.60.48.0/21;
deny 37.60.56.0/21;
deny 37.222.0.0/15;
deny 46.105.0.0/16;
deny 46.105.194.0/23;
deny 46.105.196.0/24;
deny 46.105.198.0/24;
deny 87.98.128.0/17;
deny 91.121.0.0/16;
deny 91.218.204.0/22;
deny 94.23.0.0/16;
deny 103.5.12.0/22;
deny 109.190.0.0/16;
deny 109.190.0.0/17;
deny 142.4.192.0/19;
deny 176.31.0.0/16;
deny 176.31.160.0/22;
deny 176.31.164.0/22;
deny 176.31.168.0/22;
deny 176.31.172.0/22;
deny 176.31.176.0/22;
deny 176.31.184.0/22;
deny 176.31.188.0/22;
deny 178.32.0.0/15;
deny 178.32.132.0/24;
deny 178.32.133.0/24;
deny 178.32.134.0/24;
deny 178.32.135.0/24;
deny 178.236.224.0/20;
deny 185.7.240.0/22;
deny 185.10.17.0/24;
deny 185.12.32.0/23;
deny 188.165.0.0/16;
deny 192.95.0.0/18;
deny 193.84.187.0/24;
deny 193.104.19.0/24;
deny 193.104.56.0/24;
deny 193.109.63.0/24;
deny 193.200.52.0/23;
deny 194.50.82.0/24;
deny 195.43.138.0/24;
deny 195.60.164.0/23;
deny 195.110.30.0/23;
deny 195.246.232.0/23;
deny 198.27.64.0/18;
deny 198.50.128.0/17;
deny 198.100.144.0/20;
deny 198.245.48.0/20;
deny 213.186.32.0/19;
deny 213.251.128.0/18;

# Hetzner
deny 5.9.0.0/16;
deny 46.4.0.0/16;
deny 78.46.0.0/15;
deny 85.10.192.0/18;
deny 88.198.0.0/16;
deny 144.76.0.0/16;
deny 176.9.0.0/16;
deny 178.63.0.0/16;
deny 185.12.64.0/22;
deny 188.40.0.0/16;
deny 213.133.96.0/19;
deny 213.239.192.0/18;


It's difficult to find a definitive list for these culprits, so I hope it helps anybody who may be searching for it.

ps. I've never done such a large scale ban before, so let me know if you notice any errors :)

edit: Just to add: These were either spambots or some kind of search bots/SEO tools.

lucy24




msg:4561947
 7:29 am on Apr 6, 2013 (gmt 0)

37.60.48.0/21 + 37.60.56.0/21 = 37.60.48.0/20

I had to laugh as I looked over this list. My own IP charts are color-coded, so as I scan for the next number, I know I'm getting close when I see that familiar shade of deep puke green* coming up :)

deny 46.105.0.0/16;
deny 46.105.194.0/23;
deny 46.105.196.0/24;
deny 46.105.198.0/24;

Oops! Forgot some housekeeping there.

deny 109.190.0.0/16;
deny 109.190.0.0/17;

And here.


* Not to be confused with barf yellow, which is China, grey-green, which is garden-variety robots, or muted forest green, which is RIPE.

keyplyr




msg:4561991
 9:04 am on Apr 6, 2013 (gmt 0)

Thanks Andem. I didn't have a couple of the OVH server ranges.

Some of the OVH ranges however (example the DSL ranges or the SMTP company) I don't consider a categorical threat. Also, just because OVH (or anyone) manages/owns a range, that's not enough for me to block it, unless the range is specifically used for web servers, colos, data centers, etc. If they are leasing the range to a private company (that is not a host, colo, data center, etc) I need to see bad behavior from that company before I block the range.

Had all of the Hetzner.

dstiles




msg:4562190
 8:37 pm on Apr 6, 2013 (gmt 0)

Lucy - am I missing something or is there a degree of tautology in your numbers? IE 46.105.0.0/16 covers the whole 46.105 range, ditto 109.190.0.0/16 covers the /17 range.

wilderness




msg:4562398
 5:40 pm on Apr 7, 2013 (gmt 0)

180SERVERS-1 69.194.224.0 - 69.194.239.255 69.194.224.0/20

They only have one of their own servers, however other ranges are provided via backbones.
The backbones links resulted in another server:

Colo4, LLC
COLO4-BLK7 173.237.128.0 - 173.237.191.255 173.237.128.0/18
COLO4-BLK6 174.136.0.0 - 174.136.63.255 174.136.0.0/18
COLO4-BLK1 206.123.64.0 - 206.123.127.255 206.123.64.0/18
COLO4-BLK4 207.210.192.0 - 207.210.255.255 207.210.192.0/18
COLO4-BLK5 65.99.192.0 - 65.99.255.255 65.99.192.0/18
COLO4-BLK3 72.29.96.0 - 72.29.127.255 72.29.96.0/19
COLO4-BLK2 72.249.0.0 - 72.249.191.255 72.249.128.0/18 72.249.0.0/17
COLO4-IPV6-BLK1 2607:FDB8:: - 2607:FDB8:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

180SERVER also uses NETRIPLEX, however there are already multiple threads on NETRIPLEX.

BTW, this 180server followed a denied request from Class A (88), which I'm assuming is a Farms as well:
88.114.0.0 - 88.115.255.255
netname: ELISA-LAAJAKAISTA
role: Elisa Hostmaster

dstiles




msg:4562408
 6:06 pm on Apr 7, 2013 (gmt 0)

I have 88.114.0.0/15 as a wider DSL range...

88.112.0.0/14
Elisa Oyj

Was there a proxy involved anywhere? I often find DSL IPs proxying through servers (which may or may not be ok depending on the server's function) and servers proxying through DSL IPs which is a definite no-no and is usually someone trying to scrape by pretending to be a person.

This 92 message thread spans 4 pages: 92 ( [1] 2 3 4 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved