homepage Welcome to WebmasterWorld Guest from 54.146.175.204
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 6 [7] 8 9 10 11 > >     
Server Farms - Sept. 2013
Ongoing Hosting Data Center Discussion
incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 11:55 pm on Sep 4, 2013 (gmt 0)

Continuation of the May 2013 thread:
[webmasterworld.com...]

 

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 8:59 am on Dec 13, 2013 (gmt 0)

Baidu caught today trying to crawl our site through an EU backdoor:
185.10.104.0 - 185.10.107.255 185.10.104.0/22

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 10:06 am on Dec 13, 2013 (gmt 0)

Don't know what's "backdoor" about it. All the major SEs have server ranges abroad.

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 11:17 am on Dec 13, 2013 (gmt 0)

Baidu is comprehensively blocked at our front door, so they tried to gain access via an EU server farm, whilst not identifying itself, nor visiting robots.txt.

You may encourage such behaviour: We don't.

bobothecat2



 
Msg#: 4607411 posted 4:16 pm on Dec 13, 2013 (gmt 0)

Another range to add for InternetNamesForBusiness

[webmasterworld.com...]

207.217.125.0 - 207.217.125.255 207.217.125.0/24

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 6:16 pm on Dec 13, 2013 (gmt 0)

If there's a larger Earthlink range that is geared towards colo, I'd like to be aware of it.
Anybody know?

This Class B from the new InternetNames range that bobo supplied:

207.217.0.0 - 207.217.255.255
CIDR207.217.0.0/16
NameEARTHLINK-CIDR

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4607411 posted 6:24 pm on Dec 13, 2013 (gmt 0)

via an EU server farm

It isn't a server farm. Well, it might be part of one physically, but the whole /22 belongs to Baidu Hong Kong. And you won't find anything bigger than that in 185. The whole sector started to get allocated after RIPE decreed that it was no longer going to give out IPv4 addresses in anything larger than /22 blocks. (Got a vague idea this announcement came about a year ago, but I may have misplaced a year or more.) So far they're up to, I think, the mid-30s.

It's possible that there will be some huge surge in IPv6 connectivity and then they'll drop the /22 restriction, so further along in 185 we'll start seeing bigger chunks. But that would have to happen pretty soon, or not at all.

bobothecat2



 
Msg#: 4607411 posted 8:30 pm on Dec 13, 2013 (gmt 0)

If there's a larger Earthlink range that is geared towards colo, I'd like to be aware of it.
Anybody know?


I have the following for their cloud hosting, but I'm sure it's no where near a complete list:

EarthLink Cloud Hosting aka LogicalSolutions

67.219.32.0 - 67.219.63.255 67.219.32.0/19
108.59.240.0 - 108.59.255.255 108.59.240.0/20
216.36.48.0 - 216.36.63.255 216.36.48.0/20
216.224.128.0 - 216.224.191.255 216.224.128.0/18

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 9:10 pm on Dec 13, 2013 (gmt 0)

Many thanks bobo.

I've some subnet searches (no longer possible) saved from 2005 & 2006 offering extensive ranges.

Some of those are named with "DED" and "DA" nets, which I assume to be short for dedicated & data.

Here's three more ranges to add:
EARTHLINK-BUSINESS 208.78.152.0 - 208.78.155.255 208.78.152.0/22
EARTHLINK-BUSINESS 208.83.24.0 - 208.83.31.255 208.83.24.0/21
EARTHLINK-BUSINESS 209.63.126.0 - 209.63.127.255 209.63.126.0/23

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 11:24 pm on Dec 13, 2013 (gmt 0)

Lucy: As far as we're concerned, the Low Counties have switched from carrots and Friesians to farming slippery servers: More EU toxicity.

bobothecat2



 
Msg#: 4607411 posted 1:13 pm on Dec 14, 2013 (gmt 0)

Found another range for InternetNamesForBusiness

149.115.0.0 - 149.115.255.255 149.115.0.0/16

... and another for EarthLink Cloud Hosting aka LogicalSolutions

204.8.120.0 - 204.8.127.255 204.8.120.0/21

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4607411 posted 10:47 pm on Dec 14, 2013 (gmt 0)

Anyone got a current list for Serverel? I'd never heard of them until I found a googlebot spoofer (with fake referer, no less) in another site's logs.

Forums search turns up this mention [webmasterworld.com] * for 109.206.163.nnn which turns out to be 109.206.160.0/19 in the Netherlands. Mine was 216.172.48.0/20 from the US.


* Also a phenomenal number of typos for "several" ;)

bobothecat2



 
Msg#: 4607411 posted 11:16 pm on Dec 14, 2013 (gmt 0)

Anyone got a current list for Serverel?


Ask an you shall receive...

31.148.244.0 - 31.148.247.255 31.148.244.0/22
62.122.168.0 - 62.122.175.255 62.122.168.0/21
62.182.160.0 - 62.182.167.255 62.182.160.0/21
79.110.64.0 - 79.110.79.255 79.110.64.0/20
79.110.96.0 - 79.110.111.255 79.110.96.0/20
79.110.208.0 - 79.110.239.255 79.110.208.0/20. 79.110.224.0/20
91.227.144.0 - 91.227.147.255 91.227.144.0/22
91.230.216.0 - 91.230.217.255 91.230.216.0/23
91.229.94.0 - 91.229.94.255 91.229.94.0/24
93.171.202.0 - 93.171.203.255 93.171.202.0/23
93.171.216.0 - 93.171.217.255 93.171.216.0/23
94.231.192.0 - 94.231.207.255 94.231.192.0/20
95.47.138.0 - 95.47.138.255 95.47.138.0/24
109.206.160.0 - 109.206.191.255 109.206.160.0/19
146.120.152.0 - 146.120.167.255 146.120.152.0/21, 146.120.160.0/21
146.120.228.0 - 146.120.235.255 146.120.228.0/22, 146.120.232.0/22
162.221.224.0 - 162.221.227.255 162.221.224.0/22
173.214.240.0 - 173.214.255.255 173.214.240.0/20
192.133.136.0 - 192.133.143.255 192.133.136.0/21
199.182.160.0 - 199.182.167.255 199.182.160.0/21
213.109.144.0 - 213.109.159.255 213.109.144.0/20
213.109.192.0 - 213.109.207.255 213.109.192.0/20
216.172.48.0 - 216.172.63.255 216.172.48.0/20

I'm sure there's probably a few others.

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 3:31 pm on Dec 15, 2013 (gmt 0)

As an aside, rather tha a new thread... Abbreviating CIDRs

In my .htaccess I tried abbreviating CIDRs to save "bulk"

ie. using 200/8 instead of 200.0.0.0/8

And soon reverted, because the short form doesn't cover the proper full form.

Is there a valid way of shortening CIDRs?

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 4:04 pm on Dec 15, 2013 (gmt 0)

Is there a valid way of shortening CIDRs?


Yeah!
Just remove all the denys and let everybody in ;)

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 4:07 pm on Dec 15, 2013 (gmt 0)

New (to me) Leaseweb range:

37.48.64.0 - 37.48.127.255
37.48.64.0/18
NL - Netherlands

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 4:29 pm on Dec 15, 2013 (gmt 0)

"Yeah!
Just remove all the denys and let everybody in ;)"

One on every bus.

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 4:31 pm on Dec 15, 2013 (gmt 0)

You've been letting 37. in?!

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 5:36 pm on Dec 15, 2013 (gmt 0)

Angonasec - As previously noted: I do not ban complete /8 ranges, otherwise I would have blocked several large USA ranges before now! There are several acceptable DSL ranges within 37/8 including my own country UK. For example, there is a T-Mobile range immediately below the leaseweb one.

Serverel - I didn't have a lot of those ranges listed BUT I found a strong connection between the Czech company AlfaTelecom and Serverel. In many case serverel are either a small subrange of alfatelecom or are maintained by them. In some cases (US) alfatelecom do not seem to be involved (at least not in DNS) and in Europe alfatelecom ranges are not entirely serverel by a long stretch, Of the alfatelecom ranges I have, only one is unequivacably DSL/WiFi. The others I have all have at least some server-like activity and have been blocked as such. My two lists are...

AlfaTelecom (includes some serverel sub-ranges)
92.38.0.0 - 92.38.127.255 (Teplice ADSL and Wi-FI Pool)
93.170.0.0 - 93.171.255.255
95.46.0.0 - 95.47.255.255
146.120.0.0 - 146.120.255.255
213.109.144.0 - 213.109.159.255

Serverel (European and US)
31.148.128.0 - 31.148.255.255
62.122.168.0 - 62.122.175.0
62.182.160.0 - 62.182.167.255
79.110.64.0 - 79.110.79.255
79.110.96.0 - 79.110.111.255
79.110.208.0 - 79.110.223.255
91.227.144.0 - 91.227.147.255
91.229.94.0 - 91.229.94.255
91.230.216.0 - 91.230.217.255
94.231.192.0 - 94.231.207.255
109.206.160.0 - 109.206.191.255
162.221.224.0 - 162.221.227.255
173.214.240.0 - 173.214.255.255
192.133.136.0 - 192.133.143.255
199.182.160.0 - 199.182.167.255
213.109.192.0 - 213.109.207.255
216.172.48.0 - 216.172.63.255

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 7:57 pm on Dec 15, 2013 (gmt 0)

Thanks for the ranges

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4607411 posted 10:33 pm on Dec 15, 2013 (gmt 0)

Is there a valid way of shortening CIDRs?


Deny from 200.0.0.0/8
=
Deny from 200

Anything ending in 8n can be cut short:

a.b.0.0/16
=
a.b

a.b.c.0/24
=
a.b.c

Unfortunately does not work with IPv6; they've got a different syntax.

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 11:51 pm on Dec 15, 2013 (gmt 0)

dstiles:" There are several acceptable DSL ranges within 37/8 including my own country UK."

With a severely polluted block, such as the 37. (and many other EU zones) we find it less fiddly to block all, then Allow the acceptable niches. (Of which there are few.)

Lucy24: "Deny from 200.0.0.0/8
=
Deny from 200"

Thank you :) I recall trying that approach a few months ago, but quickly reverted as it wasn't covering the cidr like the full notation does. It seems the full-stop is vital, as for example:

deny from 37. which does the job.

But apparently, these variations don't (in root .htaccess):

deny from 37/8

deny from 37./8

The Apache docs only mention the full Monty.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 7:02 pm on Dec 16, 2013 (gmt 0)

1st time I've seen hits from this data center:

CorporateColo
68.64.160.0/20
68.64.160.0 - 68.64.175.255

Inside the range of our old favovite:

PacketExchange
68.64.128.0/18
68.64.128.0 - 68.64.191.255

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4607411 posted 11:13 pm on Dec 16, 2013 (gmt 0)

Found while looking for something else:

Ecatel
80.82.64.0/20
93.174.88.0/21

Apparently a Dutch server farm and colo. Wasn't someone just saying something about... Oh, never mind.

Anyone got a full list?

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 1:58 pm on Dec 17, 2013 (gmt 0)

Not a full list; but a start, which also tames several cheeky chappies from Belarus:

79. 80. 81. 83. 85. 86. 87. 88. 93. 94. 95.

Just don't tell dstiles eh :)

Then, of course, "Allow from" your civilised visitors.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 8:17 pm on Dec 17, 2013 (gmt 0)

Blocking the whole of Europe and UK will not suit nost people. Just as me blocking all of US, South America and Asia will not solve anything. :)

Lucy - I have one extra, full list of three being...

80.82.64.0 - 80.82.79.255
89.248.160.0 - 89.248.175.255
93.174.88.0 - 93.174.95.255

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4607411 posted 9:01 pm on Dec 17, 2013 (gmt 0)



Easy to get "block 'em" thinking instead of finding more creative strategies.

bobothecat2



 
Msg#: 4607411 posted 10:09 pm on Dec 17, 2013 (gmt 0)

Lucy - I have one extra, full list of three being...

80.82.64.0 - 80.82.79.255
89.248.160.0 - 89.248.175.255
93.174.88.0 - 93.174.95.255


Please allow me to add another (Ecatel):

94.102.48.0 - 94.102.63.255

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 3:26 am on Dec 18, 2013 (gmt 0)

@dstiles More false assumptions. Apparently you missed this line;

"Then, of course, "Allow from" your civilised visitors."

It's a different approach: Bottle the nasties; and then give the trout a ladder to swim upstream :)

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 3:33 am on Dec 18, 2013 (gmt 0)

KeyP: "Easy to get "block 'em" thinking instead of finding more creative strategies."

Indeed, but there's greater satisfaction in the latter, I think you'll find.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4607411 posted 10:49 pm on Dec 18, 2013 (gmt 0)

bobothecat2 - thanks. I had the range blocked but from so far back it was before I got the ecatel label. :)

Angonasec - my policy is: if a range doesn't bothers me I let its users have access. I do not really care where they come from - china, ukraine, even - so long as there is no bad intention or action. Only if a range proves "bad" (or it's a server farm) does it get blocked. That way my clients (the people who pay me!) can get traffic from the, er, "world" wide internet. @)

Angonasec

10+ Year Member



 
Msg#: 4607411 posted 11:52 am on Dec 19, 2013 (gmt 0)

dstiles: Over 60% of web traffic in 2013 was bots, and a large proportion of that 60% was miscreant. As an avid log-watcher I'm sure you've noticed :)

Only 40% is human traffic, and not all the 40% is virtuous.

The see-saw has tipped.

The gyro toppled.

Nerd-dom and geek-dom beckon.

So those of us who do not wish to become either, eagerly await a better filtering solution than any of us has so far employed.

Meanwhile, keep bailing folks!

A Merry Christmas to all our readers.

This 327 message thread spans 11 pages: < < 327 ( 1 2 3 4 5 6 [7] 8 9 10 11 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved