homepage Welcome to WebmasterWorld Guest from 54.227.77.237
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 [6] 7 8 9 10 11 > >     
Server Farms - Sept. 2013
Ongoing Hosting Data Center Discussion
incrediBILL




msg:4607413
 11:55 pm on Sep 4, 2013 (gmt 0)

Continuation of the May 2013 thread:
[webmasterworld.com...]

 

keyplyr




msg:4625688
 8:02 pm on Nov 24, 2013 (gmt 0)


Been getting a half dozen daily single page hits, no other files, from Hewlett-Packard. These pages are not being harvested from any social sites AFAIK.

IP: 15.185.110.49
UA: Mozilla/5.0 (Windows NT 6.0; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0
ASN: AS6301 HP-CLOUD-SERVICES
15.0.0.0 - 15.255.255.255
15.0.0.0/8

Anyone know is HP sends any human traffic? Anyone blocking this range?

lucy24




msg:4625703
 9:17 pm on Nov 24, 2013 (gmt 0)

There is, or used to be, some kind of archiver (the kind whose full UA is "Mozilla/4.0 (compatible;)") at 15.195.185.75. Looking it up now, I find
bbnwebdproxy4.europe.hp.net
(emphasis mine). Free lookup doesn't offer anything similarly interesting at .185. --but who knows what hp employees get up to in their free time.

Angonasec




msg:4625758
 5:09 am on Nov 25, 2013 (gmt 0)

"half dozen daily single page hits"

If same pages each day, consider ODP type listing from the bygone era, Head call check. Otherwise, I'd raise the drawbridge, especially at the mention of Cloud.

I also suspect a portion of HP employees may be from the Wild West.

not2easy




msg:4627047
 4:23 pm on Dec 1, 2013 (gmt 0)

I am running into new DigitalOcean ranges showing up in Nov. logs:
146.185.0.0 - 146.185.255.255 146.185.0.0/16
162.243.0.0 - 162.243.255.255 162.243.0.0/16
192.241.128.0 - 192.241.255.255 192.241.128.0/17

dstiles




msg:4627071
 8:13 pm on Dec 1, 2013 (gmt 0)

Digitalocean is only part of that range (see below) but it is a generally dodgy /16.

For digitalocean I have...

37.139.0.0 - 37.139.31.255
82.196.0.0 - 82.196.15.255
146.185.128.0 - 146.185.135.255
162.243.0.0 - 162.243.255.255
185.14.184.0 - 185.14.187.255
192.34.56.0 - 192.34.63.255
192.81.208.0 - 192.81.223.255
192.241.128.0 - 192.241.255.255
198.199.64.0 - 198.199.127.255
198.211.96.0 - 198.211.127.255
208.68.36.0 - 208.68.39.255

not2easy




msg:4627104
 2:07 am on Dec 2, 2013 (gmt 0)

You're right, dstiles, just checked the reading and the IP I posted was from Arin where it was listed as Ripe-erx which is covered by Ripe.net so the numbers I posted are wrong, that was the entire range of that block. The actual IP range Ripe is reporting:
inetnum: 146.185.152.0 - 146.185.159.255
netname: DIGITALOCEAN-AMS-3
descr: Digital Ocean, Inc.
country: NL

But when I searched for a different IP in the logs (146.185.128.46) I get the range you posted:
inetnum: 146.185.128.0 - 146.185.135.255
netname: DIGITALOCEAN-AMS-3
descr: Digital Ocean, Inc.
country: NL

My first search was for 146.185.156.155 and that 146.185.152.0 - 146.185.159.255 is the result I got at Ripe.net. Wonder if any more are clustered there?

thetrasher




msg:4627243
 5:21 pm on Dec 2, 2013 (gmt 0)

whois -h whois.ripe.net -L 146.185.156.155

inetnum: 146.185.128.0 - 146.185.191.255
netname: EU-DIGITALOCEAN-20110713
descr: Digital Ocean, Inc.
country: NL

not2easy




msg:4627289
 9:13 pm on Dec 2, 2013 (gmt 0)

Well that helps, thank you thetrasher. When I'm running through log analysis at (small number) AM my digging can be less than thorough, just trying to finish and take notes. As I add things to my spreadsheet, I often see things to make a little more effort, so this saves me that trip.

dstiles




msg:4627293
 9:22 pm on Dec 2, 2013 (gmt 0)

When I check an IP range I always check (at least) one above the range and one below, to see if the range has been extended.

That is one of the few things that arin got correct: (usually) showing the full IP range. Ripe often shows sub-ranges without always giving the full range. Some records show a wider range at the bottom of the record but not always. It always pays to page down, though. :)

not2easy




msg:4627297
 9:38 pm on Dec 2, 2013 (gmt 0)

The tool I use is "Network Utility", part of iStat menus app and it does give me the full report, but I copy/paste the whole thing into a txt file until I set about adding things to my main list.
That is when things like a larger range further down the page become apparent. I haven't run across anything in here that was not verified upon closer look, it's why I appreciate it when someone points out my error. If not, I would have accepted what was pasted from the full report and not gotten the other ranges. :)

bobothecat2




msg:4627793
 1:15 pm on Dec 4, 2013 (gmt 0)

Cervalis LLC

64.238.144.0 - 64.238.159.255 64.238.144.0/20
69.176.96.0 - 69.176.111.255 69.176.96.0/20
216.244.96.0 - 216.244.127.255 216.244.96.0/19

keyplyr




msg:4628535
 4:31 am on Dec 7, 2013 (gmt 0)


AS29141.net (http://www.providerdienste.de)

80.83.120.0 - 80.83.121.255
80.83.120.0/21

Anyone have any more info? Any more ranges?

bobothecat2




msg:4628588
 12:18 pm on Dec 7, 2013 (gmt 0)

AS29141.net (http://www.providerdienste.de)

80.83.120.0 - 80.83.121.255
80.83.120.0/21

Anyone have any more info? Any more ranges?


Most of them appear to be DINETHOSTING, mixed with a few other colocation services...

62.149.192.0 - 62.149.255.255 62.149.192.0/18
79.137.224.0 - 79.137.239.255 79.137.224.0/20
80.77.172.0 - 80.77.175.255 80.77.172.0/22
89.222.192.0 - 89.222.192.255 89.222.192.0/24
89.208.144.0 - 89.208.159.255 89.208.144.0/20
92.38.192.0 - 92.38.224.255 92.38.192.0/19, 92.38.224.0/24
92.38.232.0 - 92.38.255.255 92.38.232.0/21, 92.38.240.0/20
93.90.16.0 - 93.90.31.255 93.90.16.0/20
93.188.8.0 - 93.188.15.255 93.188.8.0/21
95.163.64.0 - 95.163.127.255 95.163.64.0/18
195.14.104.0 - 195.14.105.255 195.14.104.0/23
212.113.32.0 - 212.113.39.255 212.113.32.0/21
213.248.32.0 - 213.248.47.255 213.248.32.0/20

keyplyr




msg:4628639
 5:07 pm on Dec 7, 2013 (gmt 0)

Thanks bobothecat2

dstiles




msg:4628652
 8:09 pm on Dec 7, 2013 (gmt 0)

bobothecat2 - thanks for the ranges - a few there I didn't have. :)

I have Dinet listed here under Digital Networking, which seems to cover wider ranges.

I would mention it would be advisable to recheck the ranges: I have several which extend beyond those given, sometimes well beyond.

bobothecat2




msg:4628722
 11:17 am on Dec 8, 2013 (gmt 0)

Ionity Corporation

204.11.60.0 - 204.11.63.255 204.11.60.0/22

This was the only range I could find for them. Does anyone have more?

bobothecat2




msg:4628748
 2:49 pm on Dec 8, 2013 (gmt 0)

HostMySite

65.36.128.0 - 65.36.255.255 65.36.128.0/17
66.241.192.0 - 66.241.255.255 66.241.192.0/18
67.59.128.0 - 67.59.191.255 67.59.128.0/18
76.12.0.0 - 76.12.255.255 76.12.0.0/16
204.12.0.0 - 204.12.127.255 204.12.0.0/17
208.112.0.0 - 208.112.127.255 208.112.0.0/17
209.41.160.0 - 209.41.191.255 209.41.160.0/19

wilderness




msg:4628760
 4:51 pm on Dec 8, 2013 (gmt 0)

Ionity Corporation
DAL-PNAP-02 192.110.208.0 - 192.110.215.255 192.110.208.0/21
IONITY-COM 199.19.80.0 - 199.19.83.255 199.19.80.0/22
IONITY-COM 199.192.228.0 - 199.192.231.255 199.192.228.0/22
ION-DAL01 199.231.224.0 - 199.231.227.255 199.231.224.0/22
ON-DAL02 192.65.240.0 - 192.65.243.255 192.65.240.0/22
IONITY-PNAP-DAL006 204.11.60.0 - 204.11.63.255 204.11.60.0/22
SOFNET-NETBLK-69-55-132-0-24 69.55.132.0 - 69.55.132.255 69.55.132.0/24
IONITY-COM 2605:EB00:: - 2605:EB00:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

wilderness




msg:4628762
 5:01 pm on Dec 8, 2013 (gmt 0)

more HostMySite

HOSTMYSITE 173.239.96.0 - 173.239.127.255 173.239.96.0/19
CONECTIV-63-238-164 (Qwest subnet) 63.238.164.0 - 63.238.164.255 63.238.164.0/24
WLCO-TWC02163004-LNH-HOST-MY-SITE (Level3 subnet) 64.192.141.112 - 64.192.141.127 64.192.141.112/28
EZANGA 76.12.116.128 - 76.12.116.135 76.12.116.128/29
EZANGA 76.12.116.152 - 76.12.116.159 76.12.116.152/29

dstiles




msg:4628804
 9:41 pm on Dec 8, 2013 (gmt 0)

Ionity - thanks, didn't have ANY of those - which means they are not troublesome (to me!).

HostMySite - thanks, a few I didn't have there but I think I had one or two not listed here, so my full range is currently:

64.192.141.112 - 64.192.141.127
65.36.128.0 - 65.36.255.255
65.182.184.0 - 65.182.223.255
66.241.192.0 - 66.241.255.255
67.59.128.0 - 67.59.191.255
76.12.0.0 - 76.12.255.255
173.239.96.0 - 173.239.127.0
204.12.0.0 - 204.12.127.255
208.112.0.0 - 208.112.127.255
209.41.160.0 - 209.41.191.255
216.74.0.0 - 216.74.63.255

Angonasec




msg:4628825
 12:07 am on Dec 9, 2013 (gmt 0)

Giving, normally polite, Czechs a bad name is a "for linguistic purposes" bot that ignores robots entirely;
147.250.0.0 - 147.252.255.255 147.250.0.0/15 147.252.0.0/16

dstiles




msg:4629050
 8:21 pm on Dec 9, 2013 (gmt 0)

Bit ambiguous there. That range includes several countries and I have very few IPs listed, which means I've never seen much mischief from the range.

I do have a Czech Academic IP 147.251.48.4 which has visited 9 times since March, lately with the UA...

Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); [nlp[.]fi[.]muni[.]cz...]

([] mine)

When checking DNS ranges, page down, especially on RIPE. The real information often comes at the bottom.

147.251.0.0 - 147.251.255.255
netname: MUNI-TCZ
descr: Masaryk University
country: CZ

As acedemic bots go, that is very benign. Most acedemic ranges sprout bots, usually some little student trying to show he's clever. They invariably block themselves (at least, on my system and I suspect most others here). I certainly do not block half a dozen countries because of one student bot on one IP. :)

keyplyr




msg:4629110
 11:19 pm on Dec 9, 2013 (gmt 0)

Not sure SKS-Lugan is solely a web server company. Ukraine info is sketchy:

91.200.12.0/22
91.200.12.0 - 91.200.15.255

194.79.60.0/22
194.79.60.0 - 194.79.63.255

213.111.128.0/18
213.111.128.0 - 213.111.191.255

lucy24




msg:4629119
 12:01 am on Dec 10, 2013 (gmt 0)

I certainly do not block half a dozen countries because of one student bot on one IP.

I've got 250 France,* 251 Czech R and 252 Ireland**. Has this changed? A lot of early-registration ranges are academic, one /16 per institution; the handful of 147s I've previously met generally seem to bear this out.


64.192.141.112 - 64.192.141.127

I'm getting Level3 for all of 64.192-195. Can the whole thing be safely blocked? /28 is an awfully small range for ARIN.


* Ecole Nationale Superieure Des Techniques Avancees, says free lookup (which doesn't "do" diacritics, nor yet customized capitalization).
** Dublin Institute Of Technology.

jmccormac




msg:4629137
 1:23 am on Dec 10, 2013 (gmt 0)

Unusual to see DIT mentioned. Many of the Irish academic institutions have more recent allocations.

Regards...jmcc

bobothecat2




msg:4629344
 4:07 pm on Dec 10, 2013 (gmt 0)

Carpathia Hosting

66.117.32.0 - 66.117.63.255 66.117.32.0/19
66.197.0.0 - 66.197.127.255 66.197.0.0/17
66.235.224.0 - 66.235.255.255 66.235.224.0/19
69.5.64.0 - 69.5.95.255 69.5.64.0/19
173.245.96.0 - 173.245.127.255 173.245.96.0/19
174.140.128.0 - 174.140.159.255 174.140.128.0/19
199.167.176.0 - 199.167.183.255 199.167.176.0/21
209.222.128.0 - 209.222.159.255 209.222.128.0/19
216.36.32.0 - 216.36.47.255 216.36.32.0/20

dstiles




msg:4629409
 7:58 pm on Dec 10, 2013 (gmt 0)

keyplr - There is a longish list at cleantalk[.]org/blacklists/AS35804 that gives spam stats for SKS-Lugan. The 91.200.12-15 range is prominent. So, that's two reasons to block: the spam activity (which may or may not include hacking activity) and Lucy's standby "It's UA". :) Oh, and the abuse address is vhoster[.]com which suggests virtual hosting.

194.79.60.0/22 - I do not have that blocked (I list and/or block when provoked) and there is an absence of spam shown at the URL above.

213.111.128.0/18 - there is a LOT of spam activity shown for that IP range but I have it "listed not blocked" with only a single bad IP emanating from the range since I listed the range in June 2012. That IP was a single-hit job last month.

Lucy - 147.25n - that ties in with my checks.

64.192-195 - I block selectively. For example: 64.192.0.0/22 is Unwired Broadband, Inc. which is probably OK. I have only 6 listed sub-ranges within that range and 4 of them are blocked. Of the other two there are (as yet) no further depredations.

bobothecat2 - thanks, only had three of those.

Angonasec




msg:4629484
 1:50 am on Dec 11, 2013 (gmt 0)

"As acedemic bots go, that is very benign." [sic]

Ignoring robots.txt, hit a minute, hundreds per day, benign in your book, not in mine :)

Well aware of the silly projects academia often gets up to, and experience has taught us to keep them on a tight rein.

"I certainly do not block half a dozen countries because of one student bot on one IP. :)"

Nor do we. False assumptions :)

When a CIDR trips the wire so frequently as this one has, it's a sign of shared dubious tendencies by its participants, and wiser to block it then observe the result.

bobothecat2




msg:4629643
 11:40 am on Dec 11, 2013 (gmt 0)

Another (new to me) range to add for B2 Net Solutions:

23.229.0.0 - 23.229.127.255 23.229.0.0/17

bobothecat2




msg:4629656
 12:34 pm on Dec 11, 2013 (gmt 0)

Galaxyvisions Inc

66.109.16.0 - 66.109.31.255 66.109.16.0/20
206.71.48.0 - 206.71.63.255 206.71.48.0/20
209.104.192.0 - 209.104.223.255 209.104.192.0/19
209.151.160.0 - 209.151.175.255 209.151.160.0/20

Angonasec




msg:4630360
 8:59 am on Dec 13, 2013 (gmt 0)

Baidu caught today trying to crawl our site through an EU backdoor:
185.10.104.0 - 185.10.107.255 185.10.104.0/22

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 [6] 7 8 9 10 11 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved