homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 6 7 8 [9] 10 11 > >     
Server Farms - Sept. 2013
Ongoing Hosting Data Center Discussion
incrediBILL




msg:4607413
 11:55 pm on Sep 4, 2013 (gmt 0)

Continuation of the May 2013 thread:
[webmasterworld.com...]

 

lucy24




msg:4634674
 8:22 pm on Jan 2, 2014 (gmt 0)

We've had 5. blocked for years

ALL of 5? Do you deal in geographically constrained widgets?

RIPE ranges aren't regionally apportioned. You'll find legitimate academics from the UK side by side with Ukrainian robots.

lucy24




msg:4634693
 10:19 pm on Jan 2, 2014 (gmt 0)

New one on me:

23.90.0.0/18 Serverhub Phoenix

Angonasec




msg:4634738
 2:20 am on Jan 3, 2014 (gmt 0)

Trout and eels behave differently from less desirable creatures, Lucy :)

lucy24




msg:4634760
 7:21 am on Jan 3, 2014 (gmt 0)

Color me utterly stumped.

Back to business:

Who's got a current InterNap list? Someone hereabouts said tantalizingly that they've got 18 ranges-- but didn't list them :( I've just met the one at 64.74. Well, turns out I've met it before, but the previous times didn't make an impression. (HEAD for front page. Shrug. The most recent one was considerably longer and weirder, as the duchess said to the ... Oh, never mind.)

Angonasec




msg:4634766
 8:01 am on Jan 3, 2014 (gmt 0)

Apologies Lucy, see msg:4631696

Not the most tactful time to mention cricket to an Englishman.

bobothecat2




msg:4634883
 6:45 pm on Jan 3, 2014 (gmt 0)

In2net Network Inc.

64.151.192.0 - 64.151.255.255 64.151.192.0/18
65.61.192.0 - 65.61.255.255 65.61.192.0/18
66.226.128.0 - 66.226.159.255 66.226.128.0/19
67.159.208.0 - 67.159.223.255 67.159.208.0/20
69.161.128.0 - 69.161.159.255 69.161.128.0/19
108.60.0.0 - 108.60.31.255 108.60.0.0/19
209.151.0.0 - 209.151.31.255 209.151.0.0/19

dstiles




msg:4634916
 9:59 pm on Jan 3, 2014 (gmt 0)

Internap:

63.251.0.0 - 63.251.255.255
64.7.192.0 - 64.7.223.255
64.74.0.0 - 64.74.255.255
64.94.0.0 - 64.95.255.255
66.150.0.0 - 66.151.255.255
67.63.48.0 - 67.63.63.255
69.25.0.0 - 69.25.255.255
70.42.0.0 - 70.42.255.255
72.5.0.0 - 72.5.255.255
74.201.0.0 - 74.201.255.255
74.217.0.0 - 74.217.255.255
75.98.0.0 - 75.98.95.255
77.242.192.0 - 77.242.207.255
95.172.64.0 - 95.172.95.255
207.174.0.0 - 207.174.255.255
212.118.224.0 - 212.118.255.255
216.52.0.0 - 216.52.255.255
216.133.224.0 - 216.133.255.255

bobothecat2 - thanks for the in2net ranges! :)

lucy24




msg:4635356
 10:24 pm on Jan 5, 2014 (gmt 0)

Gandi (France, hosting)
92.243.0.0/19

I can't believe they would have only the one range. But search for the name just turns up a lot of discussions about registrars. Are there alternate names?

jmccormac




msg:4635368
 10:57 pm on Jan 5, 2014 (gmt 0)

More than one range, Lucy,
They are showing as US, France and Luxembourg with 71 Class Cs showing websites in com/net/org/biz/info/mobi/asia/us/eu/de/es/fr/co/uk.

Regards...jmcc

bobothecat2




msg:4635480
 11:19 am on Jan 6, 2014 (gmt 0)

Gandi (France, hosting)
92.243.0.0/19

I can't believe they would have only the one range. But search for the name just turns up a lot of discussions about registrars. Are there alternate names?


Here's the ranges that I've been able to find:

46.226.104.0 - 46.226.111.255
92.243.0.0 - 92.243.31.255
95.142.160.0 - 95.142.175.255
173.246.96.0 - 173.246.111.255
185.26.124.0 - 185.26.127.255
217.70.176.0 - 217.70.191.255

lucy24




msg:4636258
 10:18 pm on Jan 9, 2014 (gmt 0)

192.208.176.0/20
Continuum Data Centers

I feel certain there are more, but "continuum" is not a search-friendly word ;)

Bewenched




msg:4636267
 10:56 pm on Jan 9, 2014 (gmt 0)

Had recent visits from users in this range.

107.6.0.0/16

attempting credit card fraud, but it's a server farm

keyplyr




msg:4636270
 11:55 pm on Jan 9, 2014 (gmt 0)


I feel certain there are more, but "continuum" is not a search-friendly word ;)


Continuum (aptly named)

64.79.96.0/20
64.79.96.0 - 64.79.111.255

74.119.216.0/22
74.119.216.0 - 74.119.219.255

74.121.180.0/22
74.121.180.0 - 74.121.183.255

173.243.112.0/20
173.243.112.0 - 173.243.127.255

192.208.176.0/20
192.208.176.0 - 192.208.191.255

198.50.24.0/21
198.50.24.0 - 198.50.31.255

199.96.80.0/21
199.96.80.0 - 199.96.87.255

199.192.200.0/21
199.192.200.0 - 199.192.207.255

216.107.144.0/20
216.107.144.0 - 216.107.159.255

216.231.128.0/20
216.231.128.0 - 216.231.143.255

Angonasec




msg:4636279
 12:49 am on Jan 10, 2014 (gmt 0)

On the piscine theme we present more NY small-fry;
ColoCrossing:
192.227.128.0 - 192.227.255.255 192.227.128.0/17

wilderness




msg:4636297
 2:39 am on Jan 10, 2014 (gmt 0)

ColoCrossing [whois.arin.net] not exactly "small fry".

keyplyr




msg:4636306
 3:43 am on Jan 10, 2014 (gmt 0)


Not exactly NY either.

lucy24




msg:4636311
 4:11 am on Jan 10, 2014 (gmt 0)

74.121.180.0/22
74.121.180.0 - 74.121.183.255

Ugh. Can't the /22s stay in 185 where they belong?

I've already got 74.121.184.0/21 flagged as WoWrack, whose name says it all. According to free lookup, 74.121.176.0/22 belongs to an outfit called Kontera Technologies. This sounds like someone that 98% of sites can safely block, giving you a marginally more attractive

74.121.176.0/20

My current 403 page falls all over itself apologizing: "I'm sorry, but the server thinks you are a robot."

Angonasec




msg:4636482
 3:07 pm on Jan 10, 2014 (gmt 0)

Thanks for those hints you log-junkies :)

"Colo" appears to be nudging ahead of "rack" in the black-sheep stakes.

wilderness




msg:4636518
 5:30 pm on Jan 10, 2014 (gmt 0)

For purposes of simplicity, I categorize (store) them all the same.

Colo, rack, shared hosting, VPS, Cloud and whatever else offers a general server capability.

bobothecat2




msg:4636540
 6:36 pm on Jan 10, 2014 (gmt 0)

Webhosting.Net, Inc.

38.107.84.0 - 38.107.87.255 38.107.84.0/22
63.251.155.0 - 63.251.155.255 63.251.155.0/24
64.71.224.0 - 64.71.239.255 64.71.224.0/20
66.232.40.0 - 66.232.41.255 66.232.40.0/23
66.232.56.0 - 66.232.58.255 66.232.56.0/23
66.250.216.0 - 66.250.219.255 66.250.216.0/22
67.215.160.0 - 67.215.191.255 67.215.160.0/19
69.25.50.0 - 69.25.50.255 69.25.50.0/24
69.25.176.0 - 69.25.176.255 69.25.176.0/24
162.219.244.0 - 162.219.247.255 162.219.244.0/22
173.230.224.0 - 173.230.239.255 173.230.224.0/20
199.241.103.0 - 199.241.103.255 199.241.103.0/24*
206.191.136.0 - 206.191.137.255 206.191.136.0/23
207.142.0.0 - 207.142.3.255 207.142.0.0/22
207.142.16.0 - 207.142.16.255 207.142.16.0/24
207.142.20.0 - 207.142.23.255 207.142.20.0/22
216.52.164.0 - 216.52.164.255 216.52.164.0/24


* = this is a range within a larger range for MojoHost 199.241.96.0/21 (I'll post all of theirs next).
Some of these ranges I already had blocked due to their relationship with InterNap

bobothecat2




msg:4636544
 6:47 pm on Jan 10, 2014 (gmt 0)

MojoHost

64.59.64.0 - 64.59.127.255 64.59.64.0/18
66.244.144.0 - 66.244.159.255 66.244.144.0/20
74.206.160.0 - 74.206.191.255 74.206.160.0/19
99.192.128.0 - 99.192.255.255 99.192.128.0/17
184.94.144.0 - 184.94.159.255 184.94.144.0/20
199.19.200.0 - 199.19.207.255 199.19.200.0/21
199.59.88.0 - 199.59.95.255 199.59.88.0/21
199.182.104.0 - 199.182.111.255 199.182.104.0/21
199.241.96.0 - 199.241.102.255 199.241.96.0/21
208.122.192.0 - 208.122.223.255 208.122.192.0/19

lucy24




msg:4636573
 8:54 pm on Jan 10, 2014 (gmt 0)

Colo, rack, shared hosting, VPS, Cloud and whatever else offers a general server capability.

I think he was talking about explicit name elements :)

dstiles




msg:4636575
 8:57 pm on Jan 10, 2014 (gmt 0)

My current listings include:

Continuum - 13 total:

64.79.96.0 - 64.79.111.255
69.25.250.0 - 69.25.251.255
74.117.236.0 - 74.117.239.255
74.119.216.0 - 74.119.219.255
74.121.180.0 - 74.121.183.255
74.122.192.0 - 74.122.199.255
98.50.24.0 - 98.50.31.255
173.243.112.0 - 173.243.127.255
192.208.176.0 - 192.208.191.255
198.50.24.0 - 198.50.31.255
199.96.80.0 - 199.96.87.255
199.192.200.0 - 199.192.207.255
216.107.144.0 - 216.107.159.255
216.231.128.0 - 216.231.143.255

colocrossing:

23.94.0.0 - 23.95.255.255
75.127.0.0 - 75.127.15.255
96.8.112.0 - 96.8.127.255
108.174.48.0 - 108.174.63.255
172.245.0.0 - 172.245.255.255
192.3.0.0 - 192.3.255.255
192.210.128.0 - 192.210.255.255
192.227.128.0 - 192.227.255.255
198.12.64.0 - 198.12.127.255
198.23.128.0 - 198.23.255.255
198.46.128.0 - 198.46.255.255
198.144.176.0 - 198.144.191.255
199.21.112.0 - 199.21.115.255
199.188.100.0 - 199.188.103.255
206.217.128.0 - 206.217.143.255

bobothecat2 - some ranges on your Webhosting list are sub-ranges of larger blockable ranges, but thanks for several new ranges. :)

bobothecat2




msg:4636581
 9:11 pm on Jan 10, 2014 (gmt 0)

bobothecat2 - some ranges on your Webhosting list are sub-ranges of larger blockable ranges, but thanks for several new ranges. :)


Yeah, I always try to post the actual ranges for the specific offending host/site, mainly in the event someone new is reading my post and doesn't already have the larger sub-ranges - which I'm not bound to try to post as an addition - would take way too long to try to explain. ;)

Angonasec




msg:4636633
 1:55 am on Jan 11, 2014 (gmt 0)

dstiles: "...sub-ranges of +larger blockable+ ranges"

:) = clearer message, and less fiddly.

Lucy: :)

Keep bailing, until a better solution arrives.

wilderness




msg:4636654
 8:53 am on Jan 11, 2014 (gmt 0)

Colo, rack, shared hosting, VPS, Cloud and whatever else offers a general server capability.


I think he was talking about explicit name elements


lucy,
I don't separate them by shade rather just bunch them all together for simpler reference based upon their color capability.

lucy24




msg:4636725
 7:20 pm on Jan 11, 2014 (gmt 0)

Another day, another botnet. *

107.158
ServerHub / Eonix

Things do slip between the cracks, but I think this really is a recent registration. There's a fairly hefty chunk of 107 that's still up for grabs.


* Figure of speech. This is an established botnet that I've known for months. Its pattern is: one 403 request (referer block) followed by a request for the contact page (not subect to the same referer block).

keyplyr




msg:4636744
 8:18 pm on Jan 11, 2014 (gmt 0)

I've had 107.158.0.0/16 blocked since Nov of 2013. I see occasional failed hits, but nothing big.

lucy24




msg:4637716
 11:06 pm on Jan 15, 2014 (gmt 0)

Rather than start a fresh thread...

Can someone explain in words of two syllables who or what Embedly is? Forums search comes up cold, aside from glancing mentions in threads such as the present one. Search engine leads to assorted how-tos, without ever explaining "How to what?" The URL in the UA string leaves me none the wiser. Not even the usual "If you're on this page, it's probably because you found" et cetera.

54.221.198.105 - - [14/Jan/2014:19:10:03 -0800] "GET / HTTP/1.1" 403 1068 "-" "Mozilla/5.0 (compatible; Embedly/0.2; +http:/ /support.embed.ly/)"
54.225.104.196 - - [14/Jan/2014:19:10:03 -0800] "GET /favicon.ico HTTP/1.1" 200 606 "-" "Mozilla/5.0 (compatible; Embedly/0.2; snap; +http:/ /support.embed.ly/)"

Thanks to the widely diverging IPs*, I didn't even realize they'd asked for the front page until I re-checked for this post. favicon requests get a free ride so that's all I saw.


* Ordinarily I match the first 3 sectors; this one didn't even match two. Blocked /10, of course.

bobothecat2




msg:4637718
 11:17 pm on Jan 15, 2014 (gmt 0)

Can someone explain in words of two syllables who or what Embedly is?


Amazon AWS... that's where they are coming from, which is an automatic ban from me.

keyplyr




msg:4637808
 9:06 am on Jan 16, 2014 (gmt 0)

Looks to me like a 3rd party CMS specifically to add content to a web page. Presumably audio & video media, articles & content, etc. What lead me to blocking them (aside from the AWS address which I do poke a few holes in for the occasional "exception") is this statement on the home page:

Embed turns any URL into embeddable content


IMO That implies they scrape snippets (or more) from outgoing links.

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 6 7 8 [9] 10 11 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved