homepage Welcome to WebmasterWorld Guest from 23.22.97.26
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 103 message thread spans 4 pages: < < 103 ( 1 2 [3] 4 > >     
Server Farms - May 2013
Ongoing Hosting Data Center Discussion
incrediBILL




msg:4570885
 8:14 pm on May 5, 2013 (gmt 0)

Continuation of the March 2013 thread:
[webmasterworld.com...]

 

dstiles




msg:4592573
 7:39 pm on Jul 12, 2013 (gmt 0)

I have a wider range for the datashack range. My full set of datashack is:

74.91.16.0 - 74.91.31.255
142.54.160.0 - 142.54.191.255
192.151.144.0 - 192.151.159.255
198.204.224.0 - 198.204.255.255

keyplr:
Are you sure about amernet being a cloud? I have it down as DSL and all IPs I've checked (admittedly not that many!) have closed ports.

not2easy:
My godaddy ranges currently stand at:

50.62.0.0 - 50.63.255.255
64.202.160.0 - 64.202.191.255
68.178.128.0 - 68.178.255.255
72.167.0.0 - 72.167.255.255
97.74.0.0 - 97.74.255.255
118.139.160.0 - 118.139.191.255
146.255.32.0 - 146.255.47.255
160.153.0.0 - 160.153.255.255
173.201.0.0 - 173.201.255.255
182.50.128.0 - 182.50.159.255
184.168.0.0 - 184.168.255.255
188.121.32.0 - 188.121.63.255
203.124.96.0 - 203.124.127.255
208.109.0.0 - 208.109.255.255
216.69.128.0 - 216.69.191.255

wiresix (blocked):

66.71.240.0 - 66.71.255.255
98.142.208.0 - 98.142.223.255

softlayer (blocked):

5.153.0.0 - 5.153.63.255
50.22.0.0 - 50.23.255.255
50.97.0.0 - 50.97.255.255
66.228.112.0 - 66.228.127.255
67.228.0.0 - 67.228.255.255
74.86.0.0 - 74.86.255.255
75.126.0.0 - 75.126.255.255
108.168.128.0 - 108.168.255.255
119.81.0.0 - 119.81.255.255
159.253.128.0 - 159.253.159.255
173.192.0.0 - 173.193.255.255
174.36.0.0 - 174.37.255.255
192.155.192.0 - 192.155.255.255
192.169.48.0 - 192.169.63.255
198.58.80.0 - 198.58.95.255
208.43.0.0 - 208.43.255.255
208.101.0.0 - 208.101.63.255

keyplr:

Surebroadband is actually 146.255.0.0/20 and it's UK. There are several /20 ranges in that /16 and I have three DSL and three Server in the database.

keyplyr




msg:4592628
 11:39 pm on Jul 12, 2013 (gmt 0)


keyplr:
Are you sure about amernet being a cloud? I have it down as DSL and all IPs I've checked (admittedly not that many!) have closed ports.

Their homepage lists their products as: Data, Voice, Cloud, Security and AFAICT biz connectivity (broadband/DSL.)

not2easy




msg:4592655
 3:15 am on Jul 13, 2013 (gmt 0)

Thank you @keyplyr and dstiles for the suggestions and the lists, that filled in some empty spots for me. I would not block the entire /16 range, but they were trying to access a domain that is disallowed to all bots, noindexed and has no public content. It has only one public page to say it is down for maintenance, no internal links so anything past that page gets blocked.

dstiles




msg:4592771
 2:44 pm on Jul 13, 2013 (gmt 0)

New softlayer range discovered today:

5.10.64.0 - 5.10.127.255
SoftLayer Dutch Holdings BV

Discovered because, as I watched, I was hit by ahrefs bot from Afghanistan on 16 separate IPs...

5.10.83.0 - 5.10.83.127
Ahrefs Pte Ltd

dstiles




msg:4592834
 7:16 pm on Jul 13, 2013 (gmt 0)

keyplr - sounds a bit ambiguous: could go either way.

Having said that, I've blocked two IPs from the range in the past few days for using http/1.0 but since a lot of proxy/firewalls use that it's not decisive. They may be browsing through a disguised proxy. The IPs were adjacent so it's possibly a biz group (say, /26). I could not find any open ports and being Saturday that may make sense for a biz that shuts down their system on a weekend.

keyplyr




msg:4593132
 1:46 am on Jul 15, 2013 (gmt 0)


GlobalLayer Datacenters/Colo
109.232.224.64 - 109.232.224.255
109.232.224.64/26
109.232.224.128/25

keyplyr




msg:4593140
 2:26 am on Jul 15, 2013 (gmt 0)


Mag Bross Web Services, Romania
91.213.11.0 - 91.213.11.255
91.213.11.0/24

dstiles




msg:4593325
 7:36 pm on Jul 15, 2013 (gmt 0)

keyplr - I blocked the range 109.232.224.0/21 in 2011 as Serverboost. Looks like a simple takeover/name change. With RIPE records it always pays to scroll down to the end of the record, where it usually gives the full (or at least a larger) range.

91.213.11.0/24 - I have the full /16 blocked with the note: "Seems to be "russia"-operated backbones in /24 blocks?" I had 9 /24 ranges blocked before blocking the full /16.

keyplyr




msg:4593329
 7:58 pm on Jul 15, 2013 (gmt 0)

Thanks dstiles :)

wilderness




msg:4593347
 9:26 pm on Jul 15, 2013 (gmt 0)

keyplr - I blocked the range 109.232.224.0/21 in 2011 as Serverboost. Looks like a simple takeover/name change. With RIPE records it always pays to scroll down to the end of the record, where it usually gives the full (or at least a larger) range.


It's much easier just block the entire Class A ;)

lucy24




msg:4593367
 10:35 pm on Jul 15, 2013 (gmt 0)

Easy for you to say. Your widgets aren't exactly made for crossing the Atlantic, are they? :)

wilderness




msg:4593369
 10:48 pm on Jul 15, 2013 (gmt 0)

Their known for their swimming skills ;)
Some even for diving into tanks at Atlantic City.

keyplyr




msg:4593407
 12:58 am on Jul 16, 2013 (gmt 0)


217.73.208.103
crawler-93.crawler.istella.it


ENERGIT
217.73.208.0 - 217.73.215.255
217.73.208.0/2

keyplyr




msg:4593408
 1:01 am on Jul 16, 2013 (gmt 0)



CloudSigma Servers
185.12.4.0 - 185.12.7.255
185.12.4.0/22

dstiles




msg:4593519
 1:47 pm on Jul 16, 2013 (gmt 0)

A new OVH range today from Canada:

192.99.0.0 - 192.99.255.255
192.99.0.0/16

My full list (all blocked) is now...

5.39.0.0 - 5.39.127.255
5.135.0.0 - 5.135.255.255
37.59.0.0 - 37.59.255.255
37.60.48.0 - 37.60.63.255
46.105.0.0 - 46.105.255.255
77.111.192.0 - 77.111.255.255
87.98.128.0 - 87.98.255.255
91.121.0.0 - 91.121.255.255
94.23.0.0 - 94.23.255.255
109.190.0.0 - 109.190.255.255
142.4.192.0 - 142.4.223.255
176.31.0.0 - 176.31.255.255
178.32.0.0 - 178.33.255.255
178.236.224.0 - 178.236.239.255
188.165.0.0 - 188.165.255.255
192.95.0.0 - 192.95.63.0
192.99.0.0 - 192.99.255.255
198.27.64.0 - 198.27.127.255
198.50.128.0 - 198.50.255.255
198.100.144.0 - 198.100.159.255
198.245.48.0 - 198.245.63.255
213.186.32.0 - 213.186.63.255
213.251.128.0 - 213.251.191.255

Mostly France but several Canada and one Belgium.

dstiles




msg:4593522
 1:57 pm on Jul 16, 2013 (gmt 0)

keyplr - I really hope you didn't mean /2 for energit! :)

I have two bot ranges for them:

217.73.208.11 - 217.73.208.45
217.73.208.64 - 217.73.208.126

Do you have any others? I have the Energit range enabled but have seen nothing beyond those bot ranges above.

CloudSigma - I have...

31.171.240.0 - 31.171.255.255
178.22.64.0 - 178.22.71.255
185.12.4.0 - 185.12.7.255

Wilderness - how dare you block a major UK customer base? :)

wilderness




msg:4593572
 4:27 pm on Jul 16, 2013 (gmt 0)

Wilderness - how dare you


I pretty much have the Class A's denied for all the RIPE & APNIC ranges you gents been listing, thus I'm not just focusing on the UK, course I could (and did early on) ;)

keyplyr




msg:4593578
 4:45 pm on Jul 16, 2013 (gmt 0)


Wajam
198.199.14.0 - 198.199.15.255
198.199.14.0/23

keyplyr




msg:4593585
 5:02 pm on Jul 16, 2013 (gmt 0)



@dstiles, no no other energit. Never heard of them until I saw their bot yesterday.

Thanks for the OVH list. Didn't have that new one and one other.

keyplyr




msg:4593947
 6:07 pm on Jul 17, 2013 (gmt 0)

Azar-a.net, HU
91.219.236.0 - 91.219.239.255
91.219.236.0/22

motorhaven




msg:4594495
 6:25 am on Jul 19, 2013 (gmt 0)

TEKSAVVY.COM, Canada
75.119.224.0 - 75.119.255.255
75.119.224.0/19

motorhaven




msg:4594496
 6:34 am on Jul 19, 2013 (gmt 0)

Privax.com (hidemyass.com)
66.187.64.0 - 66.187.79.255
66.187.64.0/20

It's a privacy browser/vpn service, but all I've ever gotten through here is forum spammers (who failed to get past my other measures) and crawlers.

[edited by: motorhaven at 7:01 am (utc) on Jul 19, 2013]

motorhaven




msg:4594499
 6:58 am on Jul 19, 2013 (gmt 0)

WEHOSTWEBSITES.COM, USA
192.82.248.0 - 192.82.255.255
192.82.248.0/21

keyplyr




msg:4594768
 7:27 am on Jul 20, 2013 (gmt 0)



Another privacy service:

PrivActually Ltd, Sweden
194.132.32.0 - 194.132.32.255
194.132.32.0/24

dstiles




msg:4595487
 8:49 pm on Jul 22, 2013 (gmt 0)

Another leaseweb today...

5.79.64.0 - 5.79.127.255
5.79.64.0/18

My complete leaseweb list now (all blocked) is:

5.79.64.0 - 5.79.127.255
37.1.192.0 - 37.1.223.255
37.58.48.0 - 37.58.63.255
46.165.192.0 - 46.165.255.255
62.212.64.0 - 62.212.95.255
78.159.96.0 - 78.159.127.255
82.192.64.0 - 82.192.95.255
83.149.64.0 - 83.149.127.255
84.16.224.0 - 84.16.255.255
85.17.0.0 - 85.17.255.255
89.149.192.0 - 89.149.255.255
94.75.192.0 - 94.75.255.255
95.211.0.0 - 95.211.255.255
108.59.0.0 - 108.59.15.255
178.162.128.0 - 178.162.255.255
188.72.192.0 - 188.72.255.255
192.96.200.0 - 192.96.207.255
193.27.72.0 - 193.27.73.255
198.7.56.0 - 198.7.63.255
199.58.84.0 - 199.58.87.255
199.115.112.0 - 199.115.119.255
212.32.224.0 - 212.32.255.255
212.95.32.0 - 212.95.63.255
217.20.112.0 - 217.20.127.255
217.73.56.0 - 217.73.63.255

keyplyr




msg:4595514
 9:43 pm on Jul 22, 2013 (gmt 0)

Thanks dstiles, didn't have that new range and several others :)

keyplyr




msg:4595562
 12:38 am on Jul 23, 2013 (gmt 0)


Edis Colo, Poland
37.235.48.0 - 37.235.48.255
37.235.48.0/24

dstiles




msg:4595838
 8:04 pm on Jul 23, 2013 (gmt 0)

Full range for edis at that range is 37.235.48.0 - 37.235.63.255 but various countries. Haven't seen them before. Thanks.

not2easy




msg:4599878
 11:47 pm on Aug 6, 2013 (gmt 0)

iWeb seems to be expanding - or maybe I just never saw all these ranges before. I had the 184.107.0.0 - 184.107.255.255 blocked, but this week more have popped up and the visitors are most unwelcome.
"POST /%70%68%70%70%61%74%68/%70%68%70?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%6E HTTP/1.1" is not what I want my visitors to be doing.

iWeb Technologies Inc. IWEB
64.15.128.0 - 64.15.159.255 64.15.128.0/19
64.15.138.32 - 64.15.138.63 64.15.138.32/27
72.55.128.0 - 72.55.191.255 72.55.128.0/18
72.55.140.224 - 72.55.140.255 72.55.140.224/27

More from PrivateSystems Networks PRIVATE-1
67.222.0.0 - 67.222.31.255 67.222.0.0/19

IPStrada IPS (Digit Revenue IPS)
199.15.232.0 - 199.15.239.255 199.15.232.0/21

Wowrack
216.244.64.0 - 216.244.95.255 216.244.64.0/19

WholeSale Internet (PURE PULSE MEDIA)
208.110.64.0 - 208.110.95.255 208.110.64.0/19

and the new OVH range mentioned above

dstiles




msg:4600094
 7:45 pm on Aug 7, 2013 (gmt 0)

That looks like a SQL injection attack. Used to get a lot of those but not so many recently - perhaps hidden by more efficient logging.

My current range for iweb is (all blocked):

64.15.128.0 - 64.15.159.255
70.38.0.0 - 70.38.127.255
72.55.128.0 - 72.55.191.255
108.163.128.0 - 108.163.191.255
174.142.0.0 - 174.142.255.255
184.107.0.0 - 184.107.255.255
193.228.155.0 - 193.228.155.255
198.50.96.0 - 198.50.127.255
198.72.96.0 - 198.72.127.255
209.172.32.0 - 209.172.127.255

I have privatesystems included under bluehost (not sure why). My bluehost range (all blocked) is:

50.87.0.0 - 50.87.255.255
67.20.64.0 - 67.20.127.255
67.222.0.0 - 67.222.63.255
69.89.0.0 - 69.89.31.255
69.195.64.0 - 69.195.127.255
70.40.192.0 - 70.40.223.255
74.220.192.0 - 74.220.223.255
173.254.0.0 - 173.254.127.255

wowrack (all blocked):

74.121.184.0 - 74.121.191.255
192.34.108.0 - 192.34.111.255
208.115.96.0 - 208.115.127.255
216.176.176.0 - 216.176.191.255
216.244.64.0 - 216.244.95.255

wholesale (all blocked):

69.30.192.0 - 69.30.255.255
69.197.128.0 - 69.197.191.255
119.63.200.0 - 119.63.207.255
173.208.128.0 - 173.208.255.255
204.12.192.0 - 204.12.255.255
208.110.64.0 - 208.110.95.255

keyplyr




msg:4600157
 11:10 pm on Aug 7, 2013 (gmt 0)

Nice

This 103 message thread spans 4 pages: < < 103 ( 1 2 [3] 4 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved