homepage Welcome to WebmasterWorld Guest from 54.163.91.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 [4] 5 6 7 8 9 10 11 > >     
Server Farms - Sept. 2013
Ongoing Hosting Data Center Discussion
incrediBILL




msg:4607413
 11:55 pm on Sep 4, 2013 (gmt 0)

Continuation of the May 2013 thread:
[webmasterworld.com...]

 

Angonasec




msg:4619184
 4:32 am on Oct 26, 2013 (gmt 0)

Checked, and I already block all 31.
(I know, I know, but sometimes the Anne Bolyn method is useful.)

1.3.x I'm quite fond of "retro" in most things. (Gas boiler, car, computer, missus, etc) My host would have to redesign their custom panel, logs etc., so I'm content they are "still assessing the value of Apache 2."

dstiles




msg:4619375
 6:54 pm on Oct 27, 2013 (gmt 0)

Apache 2.0 was available at least in 2002. Current version is at least 2.4.6 issued 22nd July 2013. I would ask questions, given that new versions often mean new security.

Latest apache versions with vulnerabilities at...
[httpd[.]apache[.]org...]
(remove [] from dots)

Angonasec




msg:4619566
 5:04 pm on Oct 28, 2013 (gmt 0)

Yes Sir, you are correct of course. But they are a solid host, and I've been with them for ten years now, during which the site has never been offline, or hacked. I trust their judgment; though many have left, so our server is zippier.

They will upgrade Apache eventually without me nagging them.

dstiles




msg:4619629
 8:25 pm on Oct 28, 2013 (gmt 0)

Failed to find a complete up-to-date list of amazon so my current one below...

8.18.144.0 - 8.18.145.255
23.20.0.0 - 23.23.255.255
27.0.0.0 - 27.0.3.255
46.51.128.0 - 46.51.255.255
46.137.0.0 - 46.137.255.255
50.16.0.0 - 50.19.255.255
50.112.0.0 - 50.112.255.255
54.192.0.0 - 54.255.255.255
67.202.0.0 - 67.202.63.255
72.21.192.0 - 72.21.223.255
72.44.32.0 - 72.44.63.255
75.101.128.0 - 75.101.255.255
79.125.0.0 - 79.125.127.255
87.238.80.0 - 87.238.87.255
103.4.8.0 - 103.4.15.255
107.20.0.0 - 107.23.255.255
122.248.192.0 - 122.248.255.255
174.129.0.0 - 174.129.255.255
175.41.128.0 - 175.41.255.255
176.32.64.0 - 176.32.127.255
176.34.0.0 - 176.34.255.255
177.71.128.0 - 177.71.255.255
178.236.0.0 - 178.236.15.255
184.72.0.0 - 184.73.255.255
184.169.128.0 - 184.169.255.255
199.255.192.0 - 199.255.195.255
204.236.128.0 - 204.236.255.255
205.251.192.0 - 205.251.255.255
207.171.160.0 - 207.171.191.255
216.182.224.0 - 216.182.239.255

Angonasec




msg:4619933
 2:55 am on Oct 30, 2013 (gmt 0)

Yes, a good time to update AWS blocks these new ones hit me today:
Amazon Technologies Inc. AMAZON-2011L 54.192.0.0 - 54.207.255.255 = 54.192.0.0/12
Amazon.com, Inc. AMAZO-ZIAD7 54.204.0.0 - 54.205.255.255 = 54.204.0.0/15

lucy24




msg:4619966
 5:11 am on Oct 30, 2013 (gmt 0)

Speaking of updating blocks... Is there a definitive Hetzner list? The last discussion I can find is from pretty exactly a year ago [webmasterworld.com].

I was looking up a robot, found that I'd already flagged the range* as Hetzner (why wasn't it already blocked in that case? Oh, who knows), and realized I'm overdue for a systematic listing.


* 144.76, nothing special.

bhukkel




msg:4619975
 6:32 am on Oct 30, 2013 (gmt 0)

Hetzner list according to the routing information (AS24940):

5.9.0.0/16 (DE flag)Hetzner Online AG
46.4.0.0/16 (DE flag)Hetzner Online AG
78.46.0.0/15 (DE flag)Hetzner Online AG
85.10.192.0/18 (DE flag)Hetzner Online AG
88.198.0.0/16 (DE flag)Hetzner Online AG
91.220.49.0/24 (DE flag)Simon & Stolle GbR
91.233.8.0/22 (DE flag)ITfM GmbH
144.76.0.0/16 (DE flag)Server Block
176.9.0.0/16 (DE flag)Hetzner Online AG
176.102.168.0/21 (DE flag)ITfM GmbH
178.63.0.0/16 (DE flag)Hetzner Online AG
185.12.64.0/22 (DE flag)Hetzner Online AG
188.40.0.0/16 (DE flag)hetzner Africa
193.25.170.0/23 (DE flag)ITfM GmbH
193.110.6.0/23 (DE flag)ZDIS GmbH
193.223.77.0/24 (DE flag)DOMAINFACTORY Gmbh
194.42.180.0/22 (DE flag)Network of Online-Info Service GmbH
194.42.184.0/22 (DE flag)Network of Online-Info Service GmbH
194.145.226.0/24 (DE flag)Simon & Stolle GbR
197.242.84.0/22 (ZA flag)
213.133.96.0/19 (DE flag)Hetzner Online AG
213.239.192.0/18 (DE flag)Hetzner Online AG

dstiles




msg:4620157
 9:18 pm on Oct 30, 2013 (gmt 0)

My ACTUAL Hetzner list (sans other providers) is currently...

5.9.0.0 - 5.9.255.255
41.204.192.0 - 41.204.223.255
46.4.0.0 - 46.4.255.255
78.46.0.0 - 78.47.255.255
85.10.192.0 - 85.10.255.255
88.198.0.0 - 88.198.255.255
91.220.49.0 - 91.220.49.255
144.76.0.0 - 144.76.255.255
176.9.0.0 - 176.9.255.255
178.63.0.0 - 178.63.255.255
188.40.0.0 - 188.40.255.255
194.42.176.0 - 194.42.191.255
194.145.226.0 - 194.145.226.255
196.40.96.0 - 196.40.111.255
197.221.0.0 - 197.221.63.255
213.133.96.0 - 213.133.127.255
213.239.192.0 - 213.239.255.225

This expands some of bhukkel's ranges for other farms, which use hetzner's sub-ranges.

blend27




msg:4620287
 11:54 am on Oct 31, 2013 (gmt 0)

Grand Web Solusions:
NetRange: 205.237.88.0 - 205.237.95.255
CIDR: 205.237.88.0/21
NetName: GRANDWEB-1

Caught 205.237.88.155 faking the UA(equal sign):
=Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16

bobothecat2




msg:4620812
 9:13 pm on Nov 2, 2013 (gmt 0)

NSI Hosting

199.19.64.0 - 199.19.71.255
199.19.64.0/21

lucy24




msg:4621719
 1:11 am on Nov 7, 2013 (gmt 0)

Have I missed anyone? I had no idea there were so many separate PegTech ranges until I started cleaning up my lists :(

142.0.128-143
142.4.96-127
192.74.224-255
198.2.192-255
198.200.32-63
199.180.100-103

keyplyr




msg:4621729
 2:24 am on Nov 7, 2013 (gmt 0)


Here's my current list:

137.175.0.0/17 #PegTech 137.175.0.0 - 137.175.127.255
142.0.128.0/20 #PegTech 142.0.128.0 - 142.0.143.255
142.4.96.0/19 #PegTech 142.4.96.0 - 142.4.127.255
192.74.224.0/19 #PegTech 192.74.224.0 - 192.74.255.255
198.2.192.0/18 #PegTech 198.2.192.0 - 198.2.255.255
198.200.32.0/19 #PegTech 198.200.32.0 - 198.200.63.255
199.180.100.0/22 #PegTech 199.180.100.0 - 199.180.103.255
199.188.104.0/21 #PegTech 199.188.104.0 - 199.188.111.255

lucy24




msg:4621742
 5:30 am on Nov 7, 2013 (gmt 0)

:: detour to lookup ::

Whew. Turns out I did have them flagged, they just didn't jump out in the particular way I was looking at.

New one on me:

66.172.0.0/18
ChunkHost and/or CyberVerse. I don't know-- and don't particularly care-- which of the two is the umbrella and which is the individual robot at 66.172.27.0/24

bobothecat2




msg:4621813
 11:54 am on Nov 7, 2013 (gmt 0)

More from Cyberverse:

66.180.192.0 - 66.180.207.255 = 66.180.192.0/20
216.176.192.0 - 216.176.207.255 = 216.176.192.0/20
209.151.224.0 - 209.151.255.255 = 209.151.224.0/19

blend27




msg:4621833
 1:39 pm on Nov 7, 2013 (gmt 0)

Add one to my PSYCHZ list: [webmasterworld.com...]

NetRange: 23.91.0.0 - 23.91.31.255
CIDR: 23.91.0.0/19
NetName: PSYCHZ-NETWORKS
RegDate: 2013-08-20

IP: 23.91.2.50
UA: Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64

Missing half the headers for:

Direct hit/GET "SignUp for updates" form from NONE HTTPS self. I didnt not know that was possible on my site :)

Angonasec




msg:4621834
 1:40 pm on Nov 7, 2013 (gmt 0)

Saw a Cox search routed through an AOL proxy last week which was a "clever" bot.

"Clever", because the bot used our site-search. (No Captcha)

Not human because it assembled a huge one word term using scraped text from our files.

The AOL proxy IPs used were:
64.12.116.76 and 64.12.116.145

You may call me Peter Gabriel, but I'm now blocking these AOL proxy ranges:
64.12.96.0/19
64.12.0.0/16

not2easy




msg:4621899
 7:35 pm on Nov 7, 2013 (gmt 0)

A little late to the HSI list, here's mine:
50.30.32.0/20 HSI 50.30.32.0 - 50.30.47.255
50.115.112.0/20 HSI 50.115.112.0 - 50.115.127.255
67.213.208.0/20 HSI 67.213.208.0 - 67.213.223.255
69.4.224.0/20 HSI 69.4.224.0 - 69.4.239.255
69.64.32.0/19 HSI 69.64.32.0 - 69.64.63.255 (MJ12bot)
80.77.80.0/20 HSI 80.77.80.0 - 80.77.95.255
88.214.192.0/18 HSI 88.214.192.0 - 88.214.255.255
98.158.176.0/20 HSI 98.158.176.0 - 98.158.191.255
173.192.34.64/26 HSI 173.192.34.64 - 173.192.34.127 (Aboundex)
173.224.112.0/20 HSI 173.224.112.0 - 173.224.127.255
173.244.192.0/19 HSI 173.244.192.0 - 173.244.223.255
173.255.128.0/20 HSI 173.255.128.0 - 173.255.143.255
174.127.64.0/18 HSI 174.127.64.0 - 174.127.127.255
198.105.208.0/20 HSI 198.105.208.0 - 198.105.223.255
199.189.84.0/22 HSI 199.189.84.0 - 199.189.87.255
199.189.104.0/21 HSI 199.189.104.0 - 199.189.111.255
199.195.192.0/21 HSI 199.195.192.0 - 199.195.199.255
199.217.112.0/21 HSI 199.217.112.0 - 199.217.119.255
206.217.192.0/19 HSI 206.217.192.0 - 206.217.223.255
209.239.112.0/20 HSI 209.239.112.0 - 209.239.127.255
216.119.144.0/20 HSI 216.119.144.0 - 216.119.159.255

not2easy




msg:4621900
 7:42 pm on Nov 7, 2013 (gmt 0)

Ran across a series of instant IP changing bots from server farms around the world all requesting the same non-existant page (about 18 hits with anywhere from one to four from each IP as if it was bouncing back and forth. Most were well known but I picked up a few new ones there:

Formless Networking, LLC AO-BLK-FORMLESS-1 (NET-199-48-147-32-1)
199.48.147.32 - 199.48.147.47
199.48.147.32/28
Applied Operations, LLC APPLIEDOPS-3 (NET-199-48-144-0-1)
199.48.144.0 - 199.48.147.255
199.48.144.0/22

dstiles




msg:4621908
 8:13 pm on Nov 7, 2013 (gmt 0)

Lucy / keyplr - pegtech I have...

108.186.0.0 - 108.186.255.255
137.175.0.0 - 137.175.127.255
142.0.128.0 - 142.0.143.255
142.4.96.0 - 142.4.127.255
192.74.224.0 - 192.74.255.255
198.2.192.0 - 198.2.255.255
198.200.32.0 - 198.200.63.255
199.180.100.0 - 199.180.103.255
199.188.104.0 - 199.188.111.255

There are several short subranges (eg /26) used by china amongst that lot.

Cyberverse (with one added from above list)...

66.172.0.0 - 66.172.63.255
66.180.192.0 - 66.180.207.255
209.151.224.0 - 209.151.255.255
216.176.192.0 - 216.176.207.255

lucy24




msg:4621936
 9:37 pm on Nov 7, 2013 (gmt 0)

There are several short subranges (eg /26) used by china

That's what first caught my attention. I've started tracking multinational servers separately so I don't get them mixed up with IPs that change hands in the normal course of events. The typical PegTech block is half China. F'rinstance

:: shuffling papers ::

142.0.128 China.
142.0.129.0-63, 64-95 China.
142.0.129.96-127 US.
142.0.129.128-255 China.
142.0.130-131 China
142.0.132.0-15 China.
142.0.132.16-23 US.
142.0.132.24-31, 32-63, 64-127, 128-255 China.
142.0.133.0-127, 128-191 US
142.0.133.192/27 China.
142.0.133.224-255 US.
142.0.134.0-63 China.

... and so on through 142.0.143. Details will presumably change from week to week, but at this point who gives a ###?

World's leading search engine says

Pegasus Technologies has been developing innovative electronic tracking
solutions since 1994. Our VectorTrac stolen vehicle recovery system has been

Oh. Oops. Stolen vehicles. I thought they were tracking people.

wilderness




msg:4621975
 2:46 am on Nov 8, 2013 (gmt 0)

OrgName: Yesup Ecommerce Solutions Inc.
199.21.149.246 - - [07/Nov/2013:19:28:53 -0700] "GET / HTTP/1.1" 200 2678 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0"

YESUP-COM 198.144.144.0 - 198.144.159.255 198.144.144.0/20
YESUP-COM 199.167.136.0 - 199.167.139.255 199.167.136.0/22
YESUP-COM 199.19.92.0 - 199.19.95.255 199.19.92.0/22
YESUP-COM 199.21.148.0 - 199.21.151.255 199.21.148.0/22
YESUP-COM 199.66.88.0 - 199.66.95.255 199.66.88.0/21
YESUP-COM 2604:6880:: - 2604:6880:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

dstiles




msg:4622119
 7:53 pm on Nov 8, 2013 (gmt 0)

Only had two of those. Thanks!

bobothecat2




msg:4622165
 10:48 pm on Nov 8, 2013 (gmt 0)

Peak 10, Inc.

64.239.240.0 - 64.239.255.255 - 64.239.240.0/20
65.254.208.0 - 65.254.223.255 - 65.254.208.0/20
66.18.96.0 - 66.18.127.255 - 66.18.96.0/19
66.129.64.0 - 66.129.127.255 - 66.129.64.0/18
68.64.32.0 - 68.64.47.255 - 68.64.32.0/20
68.142.176.0 - 68.142.191.255 - 68.142.176.0/20
69.2.192.0 - 69.2.223.255 - 69.2.192.0/19
72.15.192.0 - 72.15.255.255 - 72.15.192.0/18
74.84.192.0 - 74.84.207.255 - 74.84.192.0/20
74.126.80.0 - 74.126.95.255 - 74.126.80.0/20
96.45.112.0 - 96.45.127.255 - 96.45.112.0/20
96.46.240.0 - 96.46.255.255 - 96.46.240.0/20
206.196.0.0 - 206.196.31.255 - 206.196.0.0/19
208.115.32.0 - 208.115.63.255 - 208.115.32.0/19
209.34.224.0 - 209.34.255.255 - 209.34.224.0/19
209.198.192.0 - 209.198.207.255 - 209.198.192.0/20
216.26.128.0 - 216.26.191.255 - 216.26.128.0/18
216.134.192.0 - 216.134.223.255 - 216.134.192.0/19

not2easy




msg:4622251
 7:06 pm on Nov 9, 2013 (gmt 0)

I did a search here for VIRPUS and found a few posts from back in April:msg:4566932 and msg:4566941 I bumped into another one doing naughty things:

173.0.48.0 - 173.0.63.255
173.0.48.0/20

dstiles




msg:4622261
 9:29 pm on Nov 9, 2013 (gmt 0)

Didn't have several of the peak-10 ones (which means they've never attacked me) - thanks. In return, here are two you didn't list...

67.216.160.0 - 67.216.175.255
72.35.64.0 - 72.35.95.255

virpus - my current list is...

50.115.160.0 - 50.115.175.255
173.0.48.0 - 173.0.63.255
198.167.136.0 - 198.167.143.255
198.175.124.0 - 198.175.127.255
199.119.224.0 - 199.119.227.255
199.180.128.0 - 199.180.135.255
208.89.208.0 - 208.89.215.255

dstiles




msg:4622998
 5:10 pm on Nov 13, 2013 (gmt 0)

An update to my psychz list following a couple of extra hits this week...

23.91.0.0 - 23.91.31.255
23.228.192.0 - 23.228.255.255
23.238.128.0 - 23.238.255.255
74.117.56.0 - 74.117.63.255
108.171.240.0 - 108.171.255.255
173.224.208.0 - 173.224.223.255
192.184.32.0 - 192.184.63.255
192.210.48.0 - 192.210.63.255
198.13.96.0 - 198.13.127.255
199.15.112.0 - 199.15.119.255
199.71.212.0 - 199.71.215.255
199.83.88.0 - 199.83.95.255
199.119.200.0 - 199.119.207.255
208.87.240.0 - 208.87.243.255
216.24.192.0 - 216.24.207.255
216.99.144.0 - 216.99.159.255

Angonasec




msg:4623235
 2:15 pm on Nov 14, 2013 (gmt 0)

Another favicon fascinated bot:
208.92.218.177 waveform.net bot
208.92.216.0 - 208.92.223.255
CIDR: 208.92.216.0/21

wilderness




msg:4623264
 4:06 pm on Nov 14, 2013 (gmt 0)

Waveform Technology
WAVEFORM-NET-6 199.102.68.0 - 199.102.71.255 199.102.68.0/22
WAVEFORM-NET-5 199.16.184.0 - 199.16.191.255 199.16.184.0/21
WAVEFORM-NET-1 204.11.32.0 - 204.11.35.255 204.11.32.0/22
WAVEFORM-NET-2 208.64.36.0 - 208.64.39.255 208.64.36.0/22
WAVEFORM-NET-3 208.79.208.0 - 208.79.215.255 208.79.208.0/21
WAVEFORM-NET-4 208.92.216.0 - 208.92.223.255 208.92.216.0/21
WAVEFORM-NET-NITEL-1 64.237.119.0 - 64.237.119.255 64.237.119.0/24
WAVEFORM-NET-NITEL-2 64.237.125.0 - 64.237.127.255 64.237.125.0/24 64.237.126.0/23
WAVEFORM-NET-V6-1 2604:CC00:: - 2604:CC00:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

dstiles




msg:4623358
 8:50 pm on Nov 14, 2013 (gmt 0)

Thanks. Only had a couple of those.

lucy24




msg:4623384
 9:57 pm on Nov 14, 2013 (gmt 0)

WAVEFORM-NET-NITEL-1 64.237.119.0 - 64.237.119.255 64.237.119.0/24
WAVEFORM-NET-NITEL-2 64.237.125.0 - 64.237.127.255 64.237.125.0/24 64.237.126.0/23

Are you sure about this one? I get .112.0/20, which seems more likely. I looked up some of the others too because they're awfully small ranges for ARIN, but this is the only one that came out different.

wilderness




msg:4623397
 10:57 pm on Nov 14, 2013 (gmt 0)

Are you sure about this one?


lucy,
I'm not "sure" of what my name is.

FWIW, those ranges were retrieved from ARIN Whois and if there are errors?
Please complain to ARIN.

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 [4] 5 6 7 8 9 10 11 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved