homepage Welcome to WebmasterWorld Guest from 54.225.1.70
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 6 [7] 8 9 10 11 > >     
Server Farms - Sept. 2013
Ongoing Hosting Data Center Discussion
incrediBILL




msg:4607413
 11:55 pm on Sep 4, 2013 (gmt 0)

Continuation of the May 2013 thread:
[webmasterworld.com...]

 

Angonasec




msg:4630360
 8:59 am on Dec 13, 2013 (gmt 0)

Baidu caught today trying to crawl our site through an EU backdoor:
185.10.104.0 - 185.10.107.255 185.10.104.0/22

keyplyr




msg:4630370
 10:06 am on Dec 13, 2013 (gmt 0)

Don't know what's "backdoor" about it. All the major SEs have server ranges abroad.

Angonasec




msg:4630379
 11:17 am on Dec 13, 2013 (gmt 0)

Baidu is comprehensively blocked at our front door, so they tried to gain access via an EU server farm, whilst not identifying itself, nor visiting robots.txt.

You may encourage such behaviour: We don't.

bobothecat2




msg:4630466
 4:16 pm on Dec 13, 2013 (gmt 0)

Another range to add for InternetNamesForBusiness

[webmasterworld.com...]

207.217.125.0 - 207.217.125.255 207.217.125.0/24

wilderness




msg:4630491
 6:16 pm on Dec 13, 2013 (gmt 0)

If there's a larger Earthlink range that is geared towards colo, I'd like to be aware of it.
Anybody know?

This Class B from the new InternetNames range that bobo supplied:

207.217.0.0 - 207.217.255.255
CIDR207.217.0.0/16
NameEARTHLINK-CIDR

lucy24




msg:4630493
 6:24 pm on Dec 13, 2013 (gmt 0)

via an EU server farm

It isn't a server farm. Well, it might be part of one physically, but the whole /22 belongs to Baidu Hong Kong. And you won't find anything bigger than that in 185. The whole sector started to get allocated after RIPE decreed that it was no longer going to give out IPv4 addresses in anything larger than /22 blocks. (Got a vague idea this announcement came about a year ago, but I may have misplaced a year or more.) So far they're up to, I think, the mid-30s.

It's possible that there will be some huge surge in IPv6 connectivity and then they'll drop the /22 restriction, so further along in 185 we'll start seeing bigger chunks. But that would have to happen pretty soon, or not at all.

bobothecat2




msg:4630529
 8:30 pm on Dec 13, 2013 (gmt 0)

If there's a larger Earthlink range that is geared towards colo, I'd like to be aware of it.
Anybody know?


I have the following for their cloud hosting, but I'm sure it's no where near a complete list:

EarthLink Cloud Hosting aka LogicalSolutions

67.219.32.0 - 67.219.63.255 67.219.32.0/19
108.59.240.0 - 108.59.255.255 108.59.240.0/20
216.36.48.0 - 216.36.63.255 216.36.48.0/20
216.224.128.0 - 216.224.191.255 216.224.128.0/18

wilderness




msg:4630537
 9:10 pm on Dec 13, 2013 (gmt 0)

Many thanks bobo.

I've some subnet searches (no longer possible) saved from 2005 & 2006 offering extensive ranges.

Some of those are named with "DED" and "DA" nets, which I assume to be short for dedicated & data.

Here's three more ranges to add:
EARTHLINK-BUSINESS 208.78.152.0 - 208.78.155.255 208.78.152.0/22
EARTHLINK-BUSINESS 208.83.24.0 - 208.83.31.255 208.83.24.0/21
EARTHLINK-BUSINESS 209.63.126.0 - 209.63.127.255 209.63.126.0/23

Angonasec




msg:4630564
 11:24 pm on Dec 13, 2013 (gmt 0)

Lucy: As far as we're concerned, the Low Counties have switched from carrots and Friesians to farming slippery servers: More EU toxicity.

bobothecat2




msg:4630641
 1:13 pm on Dec 14, 2013 (gmt 0)

Found another range for InternetNamesForBusiness

149.115.0.0 - 149.115.255.255 149.115.0.0/16

... and another for EarthLink Cloud Hosting aka LogicalSolutions

204.8.120.0 - 204.8.127.255 204.8.120.0/21

lucy24




msg:4630734
 10:47 pm on Dec 14, 2013 (gmt 0)

Anyone got a current list for Serverel? I'd never heard of them until I found a googlebot spoofer (with fake referer, no less) in another site's logs.

Forums search turns up this mention [webmasterworld.com] * for 109.206.163.nnn which turns out to be 109.206.160.0/19 in the Netherlands. Mine was 216.172.48.0/20 from the US.


* Also a phenomenal number of typos for "several" ;)

bobothecat2




msg:4630738
 11:16 pm on Dec 14, 2013 (gmt 0)

Anyone got a current list for Serverel?


Ask an you shall receive...

31.148.244.0 - 31.148.247.255 31.148.244.0/22
62.122.168.0 - 62.122.175.255 62.122.168.0/21
62.182.160.0 - 62.182.167.255 62.182.160.0/21
79.110.64.0 - 79.110.79.255 79.110.64.0/20
79.110.96.0 - 79.110.111.255 79.110.96.0/20
79.110.208.0 - 79.110.239.255 79.110.208.0/20. 79.110.224.0/20
91.227.144.0 - 91.227.147.255 91.227.144.0/22
91.230.216.0 - 91.230.217.255 91.230.216.0/23
91.229.94.0 - 91.229.94.255 91.229.94.0/24
93.171.202.0 - 93.171.203.255 93.171.202.0/23
93.171.216.0 - 93.171.217.255 93.171.216.0/23
94.231.192.0 - 94.231.207.255 94.231.192.0/20
95.47.138.0 - 95.47.138.255 95.47.138.0/24
109.206.160.0 - 109.206.191.255 109.206.160.0/19
146.120.152.0 - 146.120.167.255 146.120.152.0/21, 146.120.160.0/21
146.120.228.0 - 146.120.235.255 146.120.228.0/22, 146.120.232.0/22
162.221.224.0 - 162.221.227.255 162.221.224.0/22
173.214.240.0 - 173.214.255.255 173.214.240.0/20
192.133.136.0 - 192.133.143.255 192.133.136.0/21
199.182.160.0 - 199.182.167.255 199.182.160.0/21
213.109.144.0 - 213.109.159.255 213.109.144.0/20
213.109.192.0 - 213.109.207.255 213.109.192.0/20
216.172.48.0 - 216.172.63.255 216.172.48.0/20

I'm sure there's probably a few others.

Angonasec




msg:4630864
 3:31 pm on Dec 15, 2013 (gmt 0)

As an aside, rather tha a new thread... Abbreviating CIDRs

In my .htaccess I tried abbreviating CIDRs to save "bulk"

ie. using 200/8 instead of 200.0.0.0/8

And soon reverted, because the short form doesn't cover the proper full form.

Is there a valid way of shortening CIDRs?

wilderness




msg:4630872
 4:04 pm on Dec 15, 2013 (gmt 0)

Is there a valid way of shortening CIDRs?


Yeah!
Just remove all the denys and let everybody in ;)

dstiles




msg:4630874
 4:07 pm on Dec 15, 2013 (gmt 0)

New (to me) Leaseweb range:

37.48.64.0 - 37.48.127.255
37.48.64.0/18
NL - Netherlands

Angonasec




msg:4630876
 4:29 pm on Dec 15, 2013 (gmt 0)

"Yeah!
Just remove all the denys and let everybody in ;)"

One on every bus.

Angonasec




msg:4630877
 4:31 pm on Dec 15, 2013 (gmt 0)

You've been letting 37. in?!

dstiles




msg:4630883
 5:36 pm on Dec 15, 2013 (gmt 0)

Angonasec - As previously noted: I do not ban complete /8 ranges, otherwise I would have blocked several large USA ranges before now! There are several acceptable DSL ranges within 37/8 including my own country UK. For example, there is a T-Mobile range immediately below the leaseweb one.

Serverel - I didn't have a lot of those ranges listed BUT I found a strong connection between the Czech company AlfaTelecom and Serverel. In many case serverel are either a small subrange of alfatelecom or are maintained by them. In some cases (US) alfatelecom do not seem to be involved (at least not in DNS) and in Europe alfatelecom ranges are not entirely serverel by a long stretch, Of the alfatelecom ranges I have, only one is unequivacably DSL/WiFi. The others I have all have at least some server-like activity and have been blocked as such. My two lists are...

AlfaTelecom (includes some serverel sub-ranges)
92.38.0.0 - 92.38.127.255 (Teplice ADSL and Wi-FI Pool)
93.170.0.0 - 93.171.255.255
95.46.0.0 - 95.47.255.255
146.120.0.0 - 146.120.255.255
213.109.144.0 - 213.109.159.255

Serverel (European and US)
31.148.128.0 - 31.148.255.255
62.122.168.0 - 62.122.175.0
62.182.160.0 - 62.182.167.255
79.110.64.0 - 79.110.79.255
79.110.96.0 - 79.110.111.255
79.110.208.0 - 79.110.223.255
91.227.144.0 - 91.227.147.255
91.229.94.0 - 91.229.94.255
91.230.216.0 - 91.230.217.255
94.231.192.0 - 94.231.207.255
109.206.160.0 - 109.206.191.255
162.221.224.0 - 162.221.227.255
173.214.240.0 - 173.214.255.255
192.133.136.0 - 192.133.143.255
199.182.160.0 - 199.182.167.255
213.109.192.0 - 213.109.207.255
216.172.48.0 - 216.172.63.255

keyplyr




msg:4630908
 7:57 pm on Dec 15, 2013 (gmt 0)

Thanks for the ranges

lucy24




msg:4630933
 10:33 pm on Dec 15, 2013 (gmt 0)

Is there a valid way of shortening CIDRs?


Deny from 200.0.0.0/8
=
Deny from 200

Anything ending in 8n can be cut short:

a.b.0.0/16
=
a.b

a.b.c.0/24
=
a.b.c

Unfortunately does not work with IPv6; they've got a different syntax.

Angonasec




msg:4630941
 11:51 pm on Dec 15, 2013 (gmt 0)

dstiles:" There are several acceptable DSL ranges within 37/8 including my own country UK."

With a severely polluted block, such as the 37. (and many other EU zones) we find it less fiddly to block all, then Allow the acceptable niches. (Of which there are few.)

Lucy24: "Deny from 200.0.0.0/8
=
Deny from 200"

Thank you :) I recall trying that approach a few months ago, but quickly reverted as it wasn't covering the cidr like the full notation does. It seems the full-stop is vital, as for example:

deny from 37. which does the job.

But apparently, these variations don't (in root .htaccess):

deny from 37/8

deny from 37./8

The Apache docs only mention the full Monty.

keyplyr




msg:4631196
 7:02 pm on Dec 16, 2013 (gmt 0)

1st time I've seen hits from this data center:

CorporateColo
68.64.160.0/20
68.64.160.0 - 68.64.175.255

Inside the range of our old favovite:

PacketExchange
68.64.128.0/18
68.64.128.0 - 68.64.191.255

lucy24




msg:4631322
 11:13 pm on Dec 16, 2013 (gmt 0)

Found while looking for something else:

Ecatel
80.82.64.0/20
93.174.88.0/21

Apparently a Dutch server farm and colo. Wasn't someone just saying something about... Oh, never mind.

Anyone got a full list?

Angonasec




msg:4631472
 1:58 pm on Dec 17, 2013 (gmt 0)

Not a full list; but a start, which also tames several cheeky chappies from Belarus:

79. 80. 81. 83. 85. 86. 87. 88. 93. 94. 95.

Just don't tell dstiles eh :)

Then, of course, "Allow from" your civilised visitors.

dstiles




msg:4631602
 8:17 pm on Dec 17, 2013 (gmt 0)

Blocking the whole of Europe and UK will not suit nost people. Just as me blocking all of US, South America and Asia will not solve anything. :)

Lucy - I have one extra, full list of three being...

80.82.64.0 - 80.82.79.255
89.248.160.0 - 89.248.175.255
93.174.88.0 - 93.174.95.255

keyplyr




msg:4631608
 9:01 pm on Dec 17, 2013 (gmt 0)



Easy to get "block 'em" thinking instead of finding more creative strategies.

bobothecat2




msg:4631630
 10:09 pm on Dec 17, 2013 (gmt 0)

Lucy - I have one extra, full list of three being...

80.82.64.0 - 80.82.79.255
89.248.160.0 - 89.248.175.255
93.174.88.0 - 93.174.95.255


Please allow me to add another (Ecatel):

94.102.48.0 - 94.102.63.255

Angonasec




msg:4631696
 3:26 am on Dec 18, 2013 (gmt 0)

@dstiles More false assumptions. Apparently you missed this line;

"Then, of course, "Allow from" your civilised visitors."

It's a different approach: Bottle the nasties; and then give the trout a ladder to swim upstream :)

Angonasec




msg:4631698
 3:33 am on Dec 18, 2013 (gmt 0)

KeyP: "Easy to get "block 'em" thinking instead of finding more creative strategies."

Indeed, but there's greater satisfaction in the latter, I think you'll find.

dstiles




msg:4631948
 10:49 pm on Dec 18, 2013 (gmt 0)

bobothecat2 - thanks. I had the range blocked but from so far back it was before I got the ecatel label. :)

Angonasec - my policy is: if a range doesn't bothers me I let its users have access. I do not really care where they come from - china, ukraine, even - so long as there is no bad intention or action. Only if a range proves "bad" (or it's a server farm) does it get blocked. That way my clients (the people who pay me!) can get traffic from the, er, "world" wide internet. @)

Angonasec




msg:4632121
 11:52 am on Dec 19, 2013 (gmt 0)

dstiles: Over 60% of web traffic in 2013 was bots, and a large proportion of that 60% was miscreant. As an avid log-watcher I'm sure you've noticed :)

Only 40% is human traffic, and not all the 40% is virtuous.

The see-saw has tipped.

The gyro toppled.

Nerd-dom and geek-dom beckon.

So those of us who do not wish to become either, eagerly await a better filtering solution than any of us has so far employed.

Meanwhile, keep bailing folks!

A Merry Christmas to all our readers.

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 6 [7] 8 9 10 11 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved