homepage Welcome to WebmasterWorld Guest from 54.145.183.190
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 [6] 7 8 9 10 11 > >     
Server Farms - Sept. 2013
Ongoing Hosting Data Center Discussion
incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 11:55 pm on Sep 4, 2013 (gmt 0)

Continuation of the May 2013 thread:
[webmasterworld.com...]

 

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 8:02 pm on Nov 24, 2013 (gmt 0)


Been getting a half dozen daily single page hits, no other files, from Hewlett-Packard. These pages are not being harvested from any social sites AFAIK.

IP: 15.185.110.49
UA: Mozilla/5.0 (Windows NT 6.0; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0
ASN: AS6301 HP-CLOUD-SERVICES
15.0.0.0 - 15.255.255.255
15.0.0.0/8

Anyone know is HP sends any human traffic? Anyone blocking this range?

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4607411 posted 9:17 pm on Nov 24, 2013 (gmt 0)

There is, or used to be, some kind of archiver (the kind whose full UA is "Mozilla/4.0 (compatible;)") at 15.195.185.75. Looking it up now, I find
bbnwebdproxy4.europe.hp.net
(emphasis mine). Free lookup doesn't offer anything similarly interesting at .185. --but who knows what hp employees get up to in their free time.

Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4607411 posted 5:09 am on Nov 25, 2013 (gmt 0)

"half dozen daily single page hits"

If same pages each day, consider ODP type listing from the bygone era, Head call check. Otherwise, I'd raise the drawbridge, especially at the mention of Cloud.

I also suspect a portion of HP employees may be from the Wild West.

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 4:23 pm on Dec 1, 2013 (gmt 0)

I am running into new DigitalOcean ranges showing up in Nov. logs:
146.185.0.0 - 146.185.255.255 146.185.0.0/16
162.243.0.0 - 162.243.255.255 162.243.0.0/16
192.241.128.0 - 192.241.255.255 192.241.128.0/17

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 8:13 pm on Dec 1, 2013 (gmt 0)

Digitalocean is only part of that range (see below) but it is a generally dodgy /16.

For digitalocean I have...

37.139.0.0 - 37.139.31.255
82.196.0.0 - 82.196.15.255
146.185.128.0 - 146.185.135.255
162.243.0.0 - 162.243.255.255
185.14.184.0 - 185.14.187.255
192.34.56.0 - 192.34.63.255
192.81.208.0 - 192.81.223.255
192.241.128.0 - 192.241.255.255
198.199.64.0 - 198.199.127.255
198.211.96.0 - 198.211.127.255
208.68.36.0 - 208.68.39.255

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 2:07 am on Dec 2, 2013 (gmt 0)

You're right, dstiles, just checked the reading and the IP I posted was from Arin where it was listed as Ripe-erx which is covered by Ripe.net so the numbers I posted are wrong, that was the entire range of that block. The actual IP range Ripe is reporting:
inetnum: 146.185.152.0 - 146.185.159.255
netname: DIGITALOCEAN-AMS-3
descr: Digital Ocean, Inc.
country: NL

But when I searched for a different IP in the logs (146.185.128.46) I get the range you posted:
inetnum: 146.185.128.0 - 146.185.135.255
netname: DIGITALOCEAN-AMS-3
descr: Digital Ocean, Inc.
country: NL

My first search was for 146.185.156.155 and that 146.185.152.0 - 146.185.159.255 is the result I got at Ripe.net. Wonder if any more are clustered there?

thetrasher

5+ Year Member



 
Msg#: 4607411 posted 5:21 pm on Dec 2, 2013 (gmt 0)

whois -h whois.ripe.net -L 146.185.156.155

inetnum: 146.185.128.0 - 146.185.191.255
netname: EU-DIGITALOCEAN-20110713
descr: Digital Ocean, Inc.
country: NL

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 9:13 pm on Dec 2, 2013 (gmt 0)

Well that helps, thank you thetrasher. When I'm running through log analysis at (small number) AM my digging can be less than thorough, just trying to finish and take notes. As I add things to my spreadsheet, I often see things to make a little more effort, so this saves me that trip.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 9:22 pm on Dec 2, 2013 (gmt 0)

When I check an IP range I always check (at least) one above the range and one below, to see if the range has been extended.

That is one of the few things that arin got correct: (usually) showing the full IP range. Ripe often shows sub-ranges without always giving the full range. Some records show a wider range at the bottom of the record but not always. It always pays to page down, though. :)

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 9:38 pm on Dec 2, 2013 (gmt 0)

The tool I use is "Network Utility", part of iStat menus app and it does give me the full report, but I copy/paste the whole thing into a txt file until I set about adding things to my main list.
That is when things like a larger range further down the page become apparent. I haven't run across anything in here that was not verified upon closer look, it's why I appreciate it when someone points out my error. If not, I would have accepted what was pasted from the full report and not gotten the other ranges. :)

bobothecat2



 
Msg#: 4607411 posted 1:15 pm on Dec 4, 2013 (gmt 0)

Cervalis LLC

64.238.144.0 - 64.238.159.255 64.238.144.0/20
69.176.96.0 - 69.176.111.255 69.176.96.0/20
216.244.96.0 - 216.244.127.255 216.244.96.0/19

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 4:31 am on Dec 7, 2013 (gmt 0)


AS29141.net (http://www.providerdienste.de)

80.83.120.0 - 80.83.121.255
80.83.120.0/21

Anyone have any more info? Any more ranges?

bobothecat2



 
Msg#: 4607411 posted 12:18 pm on Dec 7, 2013 (gmt 0)

AS29141.net (http://www.providerdienste.de)

80.83.120.0 - 80.83.121.255
80.83.120.0/21

Anyone have any more info? Any more ranges?


Most of them appear to be DINETHOSTING, mixed with a few other colocation services...

62.149.192.0 - 62.149.255.255 62.149.192.0/18
79.137.224.0 - 79.137.239.255 79.137.224.0/20
80.77.172.0 - 80.77.175.255 80.77.172.0/22
89.222.192.0 - 89.222.192.255 89.222.192.0/24
89.208.144.0 - 89.208.159.255 89.208.144.0/20
92.38.192.0 - 92.38.224.255 92.38.192.0/19, 92.38.224.0/24
92.38.232.0 - 92.38.255.255 92.38.232.0/21, 92.38.240.0/20
93.90.16.0 - 93.90.31.255 93.90.16.0/20
93.188.8.0 - 93.188.15.255 93.188.8.0/21
95.163.64.0 - 95.163.127.255 95.163.64.0/18
195.14.104.0 - 195.14.105.255 195.14.104.0/23
212.113.32.0 - 212.113.39.255 212.113.32.0/21
213.248.32.0 - 213.248.47.255 213.248.32.0/20

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 5:07 pm on Dec 7, 2013 (gmt 0)

Thanks bobothecat2

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 8:09 pm on Dec 7, 2013 (gmt 0)

bobothecat2 - thanks for the ranges - a few there I didn't have. :)

I have Dinet listed here under Digital Networking, which seems to cover wider ranges.

I would mention it would be advisable to recheck the ranges: I have several which extend beyond those given, sometimes well beyond.

bobothecat2



 
Msg#: 4607411 posted 11:17 am on Dec 8, 2013 (gmt 0)

Ionity Corporation

204.11.60.0 - 204.11.63.255 204.11.60.0/22

This was the only range I could find for them. Does anyone have more?

bobothecat2



 
Msg#: 4607411 posted 2:49 pm on Dec 8, 2013 (gmt 0)

HostMySite

65.36.128.0 - 65.36.255.255 65.36.128.0/17
66.241.192.0 - 66.241.255.255 66.241.192.0/18
67.59.128.0 - 67.59.191.255 67.59.128.0/18
76.12.0.0 - 76.12.255.255 76.12.0.0/16
204.12.0.0 - 204.12.127.255 204.12.0.0/17
208.112.0.0 - 208.112.127.255 208.112.0.0/17
209.41.160.0 - 209.41.191.255 209.41.160.0/19

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 4:51 pm on Dec 8, 2013 (gmt 0)

Ionity Corporation
DAL-PNAP-02 192.110.208.0 - 192.110.215.255 192.110.208.0/21
IONITY-COM 199.19.80.0 - 199.19.83.255 199.19.80.0/22
IONITY-COM 199.192.228.0 - 199.192.231.255 199.192.228.0/22
ION-DAL01 199.231.224.0 - 199.231.227.255 199.231.224.0/22
ON-DAL02 192.65.240.0 - 192.65.243.255 192.65.240.0/22
IONITY-PNAP-DAL006 204.11.60.0 - 204.11.63.255 204.11.60.0/22
SOFNET-NETBLK-69-55-132-0-24 69.55.132.0 - 69.55.132.255 69.55.132.0/24
IONITY-COM 2605:EB00:: - 2605:EB00:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 5:01 pm on Dec 8, 2013 (gmt 0)

more HostMySite

HOSTMYSITE 173.239.96.0 - 173.239.127.255 173.239.96.0/19
CONECTIV-63-238-164 (Qwest subnet) 63.238.164.0 - 63.238.164.255 63.238.164.0/24
WLCO-TWC02163004-LNH-HOST-MY-SITE (Level3 subnet) 64.192.141.112 - 64.192.141.127 64.192.141.112/28
EZANGA 76.12.116.128 - 76.12.116.135 76.12.116.128/29
EZANGA 76.12.116.152 - 76.12.116.159 76.12.116.152/29

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 9:41 pm on Dec 8, 2013 (gmt 0)

Ionity - thanks, didn't have ANY of those - which means they are not troublesome (to me!).

HostMySite - thanks, a few I didn't have there but I think I had one or two not listed here, so my full range is currently:

64.192.141.112 - 64.192.141.127
65.36.128.0 - 65.36.255.255
65.182.184.0 - 65.182.223.255
66.241.192.0 - 66.241.255.255
67.59.128.0 - 67.59.191.255
76.12.0.0 - 76.12.255.255
173.239.96.0 - 173.239.127.0
204.12.0.0 - 204.12.127.255
208.112.0.0 - 208.112.127.255
209.41.160.0 - 209.41.191.255
216.74.0.0 - 216.74.63.255

Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4607411 posted 12:07 am on Dec 9, 2013 (gmt 0)

Giving, normally polite, Czechs a bad name is a "for linguistic purposes" bot that ignores robots entirely;
147.250.0.0 - 147.252.255.255 147.250.0.0/15 147.252.0.0/16

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 8:21 pm on Dec 9, 2013 (gmt 0)

Bit ambiguous there. That range includes several countries and I have very few IPs listed, which means I've never seen much mischief from the range.

I do have a Czech Academic IP 147.251.48.4 which has visited 9 times since March, lately with the UA...

Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); [nlp[.]fi[.]muni[.]cz...]

([] mine)

When checking DNS ranges, page down, especially on RIPE. The real information often comes at the bottom.

147.251.0.0 - 147.251.255.255
netname: MUNI-TCZ
descr: Masaryk University
country: CZ

As acedemic bots go, that is very benign. Most acedemic ranges sprout bots, usually some little student trying to show he's clever. They invariably block themselves (at least, on my system and I suspect most others here). I certainly do not block half a dozen countries because of one student bot on one IP. :)

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 11:19 pm on Dec 9, 2013 (gmt 0)

Not sure SKS-Lugan is solely a web server company. Ukraine info is sketchy:

91.200.12.0/22
91.200.12.0 - 91.200.15.255

194.79.60.0/22
194.79.60.0 - 194.79.63.255

213.111.128.0/18
213.111.128.0 - 213.111.191.255

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4607411 posted 12:01 am on Dec 10, 2013 (gmt 0)

I certainly do not block half a dozen countries because of one student bot on one IP.

I've got 250 France,* 251 Czech R and 252 Ireland**. Has this changed? A lot of early-registration ranges are academic, one /16 per institution; the handful of 147s I've previously met generally seem to bear this out.


64.192.141.112 - 64.192.141.127

I'm getting Level3 for all of 64.192-195. Can the whole thing be safely blocked? /28 is an awfully small range for ARIN.


* Ecole Nationale Superieure Des Techniques Avancees, says free lookup (which doesn't "do" diacritics, nor yet customized capitalization).
** Dublin Institute Of Technology.

jmccormac

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 1:23 am on Dec 10, 2013 (gmt 0)

Unusual to see DIT mentioned. Many of the Irish academic institutions have more recent allocations.

Regards...jmcc

bobothecat2



 
Msg#: 4607411 posted 4:07 pm on Dec 10, 2013 (gmt 0)

Carpathia Hosting

66.117.32.0 - 66.117.63.255 66.117.32.0/19
66.197.0.0 - 66.197.127.255 66.197.0.0/17
66.235.224.0 - 66.235.255.255 66.235.224.0/19
69.5.64.0 - 69.5.95.255 69.5.64.0/19
173.245.96.0 - 173.245.127.255 173.245.96.0/19
174.140.128.0 - 174.140.159.255 174.140.128.0/19
199.167.176.0 - 199.167.183.255 199.167.176.0/21
209.222.128.0 - 209.222.159.255 209.222.128.0/19
216.36.32.0 - 216.36.47.255 216.36.32.0/20

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 7:58 pm on Dec 10, 2013 (gmt 0)

keyplr - There is a longish list at cleantalk[.]org/blacklists/AS35804 that gives spam stats for SKS-Lugan. The 91.200.12-15 range is prominent. So, that's two reasons to block: the spam activity (which may or may not include hacking activity) and Lucy's standby "It's UA". :) Oh, and the abuse address is vhoster[.]com which suggests virtual hosting.

194.79.60.0/22 - I do not have that blocked (I list and/or block when provoked) and there is an absence of spam shown at the URL above.

213.111.128.0/18 - there is a LOT of spam activity shown for that IP range but I have it "listed not blocked" with only a single bad IP emanating from the range since I listed the range in June 2012. That IP was a single-hit job last month.

Lucy - 147.25n - that ties in with my checks.

64.192-195 - I block selectively. For example: 64.192.0.0/22 is Unwired Broadband, Inc. which is probably OK. I have only 6 listed sub-ranges within that range and 4 of them are blocked. Of the other two there are (as yet) no further depredations.

bobothecat2 - thanks, only had three of those.

Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4607411 posted 1:50 am on Dec 11, 2013 (gmt 0)

"As acedemic bots go, that is very benign." [sic]

Ignoring robots.txt, hit a minute, hundreds per day, benign in your book, not in mine :)

Well aware of the silly projects academia often gets up to, and experience has taught us to keep them on a tight rein.

"I certainly do not block half a dozen countries because of one student bot on one IP. :)"

Nor do we. False assumptions :)

When a CIDR trips the wire so frequently as this one has, it's a sign of shared dubious tendencies by its participants, and wiser to block it then observe the result.

bobothecat2



 
Msg#: 4607411 posted 11:40 am on Dec 11, 2013 (gmt 0)

Another (new to me) range to add for B2 Net Solutions:

23.229.0.0 - 23.229.127.255 23.229.0.0/17

bobothecat2



 
Msg#: 4607411 posted 12:34 pm on Dec 11, 2013 (gmt 0)

Galaxyvisions Inc

66.109.16.0 - 66.109.31.255 66.109.16.0/20
206.71.48.0 - 206.71.63.255 206.71.48.0/20
209.104.192.0 - 209.104.223.255 209.104.192.0/19
209.151.160.0 - 209.151.175.255 209.151.160.0/20

Angonasec

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4607411 posted 8:59 am on Dec 13, 2013 (gmt 0)

Baidu caught today trying to crawl our site through an EU backdoor:
185.10.104.0 - 185.10.107.255 185.10.104.0/22

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 [6] 7 8 9 10 11 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved