Server Farms - July 2015

Tracking and Reporting Data Center IP Ranges

[Ocean10000]

Continuation of the Server Farm threads.

This is where we report data center IP ranges as they are discovered or change in the rapidly evolving assigned IP landscape.

Past server farm threads:

• Feb 2015 [webmasterworld.com...]
  
  
• Aug 2014 [webmasterworld.com...]
  
  
• Apr 2014 [webmasterworld.com...]
  
  
• Feb 2014 [webmasterworld.com...]
  
  
• Sep 2013 [webmasterworld.com...]
  
  
• May 2013 [webmasterworld.com...]
  
  
• Mar 2013 [webmasterworld.com...]
  
  
• May 2012 [webmasterworld.com...]
  

[lucy24]

Must be a server config thing. I don't have any forwarding code in htaccess.

Do you have a line like this at the very beginning of your RewriteRules?

RewriteRule forbidden\.html - [L]

You don't need one for lockouts issued via "Deny from..." because your host almost certainly has a <Files> envelope in the config file to cover the standard 403 documents. But if you issue lockouts via mod_rewrite, you need an exemption for the error document there too, since nothing can be inherited. At least until all our hosts move to 2.4 with its new range of InheritDown settings. Otherwise the 403 leads to an internal request for "example.wrong/forbidden.html" that gets externally redirected to "example.right/forbidden.html" ... and then, adding to your annoyances, you'll start getting explicit requests for "forbidden.html". (This happened to me on one typo domain. It took embarrassingly long for me to figure out why all these unwanted visitors were asking for it by name.)

A 403 doesn't confirm the existence of the document (other than the 403 page) does it?

No, but I wouldn't be surprised if some robots interpret it as "there must be something they don't want me to see"; that's the reason some of them go away faster if you serve some other response, such as a redirect, 410 or 404 (manually coded as [R=404]). I had to do a manual 404 for a while because the bingbot somehow learned the name of one directory's custom 410 page and kept asking for it-- and obviously you can't say that gone.html is gone, or you get an infinite loop.

For that matter, a 403 doesn't confirm the existence of anything in particular, because unless you've goofed (as I did), the unwanted visitor will never learn the actual name and location of the document.

Edit:
It's possible you can also prevent unwanted redirects by attaching the [NS] flag to the domain-name-canonicalization redirect, but I haven't actually tested this. Will do so now.

[keyplyr]

Do you have a line like this at the very beginning of your RewriteRules?

RewriteRule forbidden\.html - [L]

Nope, I put it at the very end of my rewrite rules since I use different conditions and different 403 documents for various purposes. The 403 for mischievous humans is different than the one for bad bots. Some blocked agents are also allowed to get other files and some aren't. Example: agents that get blocked by hacker-type activity aren't allowed to get any other files. A stupid human trying to scrape content/images with M$Office gets a 403 page offering possible reasons why they got blocked. Blocked legit bots are allowed other web agent documents like this:

RewriteRule !^(apple-touch-icon(.*)?\.png|favicon\.ico|forbidden\.html|w3c/p3p\.xml|w3c/policy1\.xml|robots\.txt)$ - [F]

[edited by: keyplyr at 7:12 am (utc) on Oct 31, 2015]

[lucy24]

Gaah, you read too fast. I've just been experimenting on my test site. On my present server configuration-- which should be roughly identical to at least some of yours-- the [NS] flag does not prevent the 403 document from being redirected, thereby exposing its real name and location. (I also discovered that mod_rewrite executes before mod_dir, which should not have surprised me, since "inverse alphabetical order" is a pretty decent rough-and-ready rule. I learned this because I first tried forcing a 403 by asking for files in directories with no index page; requests to the wrong hostname get redirected before the 403 is issued.)

So if you can't have a blanket [L] at the beginning of all your RewriteRules (remember, this line applies only to your named error documents, and you can throw in as many names as you like, or even a whole directory*) you'll need an additional condition in your domain-name redirect, like

RewriteCond %{REQUEST_URI} !forbidden\.html

Blocked legit bots are allowed robots.txt and other web agent documents like this:

Almost anything that can be expressed with
!blahblah [F]
in one location, can also be done with
blahblah [L]
in a different location. But I still don't see any situation where you would want to deny access to the 403 document itself, and that's what the [L] rule is for.

The 403 for mischievous humans is different than the one for bad bots.

How does this work? You can set different ErrorDocuments for different directories, but beyond that there's not much leeway. And you can cheat by serving some unwanted visitors an entirely different response like the celebrated 418. Or did you mean only that the content of the ErrorDocument is variable because there's php stuff looking at aspects of the request?


* This is fine on my test site, where the /blahblah/ directory contains nothing but error documents. On my real site I made the mistake of using the same directory for visible documents as well, darn it all, so the [L] rules have to give specific names.

[keyplyr]

Yup same URL but the page writes itself different accoding to the conditions met. One of those If... Then... things.

[keyplyr]

BTW dstiles, I have a larger range for ServerIUS:
31.220.40.0 - 31.220.45.255
31.220.40.0/22
31.220.44.0/23

[dstiles]

More Amazon...

52.32.0.0 - 52.63.255.255
52.32.0.0/11

I thought the world was running short of ipv4 IPs?

[dstiles]

keyplr - thanks. I have 31.220.40.0/22 as HostHatch (USA), no mention of Severius; but I can believe it. :)

[lucy24]

I thought the world was running short of ipv4 IPs

That just means that if you were canny enough to pick up an /8 back when they were throwing them around like confetti, you can now name your price. In addition to the big corporate ones (Merck, duPont etc) there are at least two UK governmental departments that each have an allocated /8 when it's wildly unlikely they will need more than /12 of it in anyone's lifetime. I think there's also a Swiss one. Also one or two in France, but somehow I don't see the French government selling off anything that could be of benefit to anyone else, whether or not they need it themselves.

[keyplyr]

keyplr - thanks. I have 31.220.40.0/22 as HostHatch (USA), no mention of Severius; but I can believe it. :)

My look-up tool shows 31.220.40.0/23 as Serverius and /22 also as HostHatch but says [e-mail] is tech support, so...

More Amazon...
52.32.0.0 - 52.63.255.255
52.32.0.0/11

Looks like they've had it a while (registered 2011-12-08) but may have recently started using it.

[keyplyr]

So AWS 52s are:
52.0.0.0/11
52.0.0.0 - 52.31.255.255
52.32.0.0/11
52.32.0.0 - 52.63.255.255
52.64.0.0/11
52.64.0.0 - 52.95.255.255

Which is:
52.0.0.0 - 52.95.255.255
52.0.0.0/10
52.64.0.0/11

Anyone aware of more AWS 52s?

[dstiles]

New-ish GoDadddy...

198.71.128.0 - 198.71.255.255
198.71.128.0/17

[lucy24]

Are there humans within
31.220.0.0/20
? Looking up a botnet from 31.220.6.abc I met a tangle of names-- TerraTransit, HostHatch, Portlane-- along with a tangle of countries-- US, Germany, Netherlands (no Sweden, in spite of the Portlane). This doesn't inspire me with confidence.

[keyplyr]

I agree, it's a hodgepodge. I currently only block the lower end:
31.220.0.0 - 31.220.45.255
I have not noticed humans in the rest of it, but ya never know.

terratransit.de looks like the fiber backbone.

[lucy24]

31.220.0.0 - 31.220.45.255

Typo for ... something? The range I quoted (/20 i.e. 0-15) is smaller than this, and 45 is a weird number*.


* In, ahem, the CIDR context. 3*3*5 is perfectly fine.

[keyplyr]

Not to imply that a single CIDR does the job:)

I'm actually blocking:
31.220.0.0/22 #Koddos 31.220.0.0 - 31.220.3.255
31.220.6.0/24 #HostHatch 31.220.6.0 - 31.220.6.255
31.220.7.0/24 #HostHatch 31.220.7.0 - 31.220.7.255
31.220.29.0/24 #AL-Albanian-Hosting 31.220.29.0 - 31.220.29.255
31.220.30.0/24 #HostHatch 31.220.30.0 - 31.220.30.255
31.220.40.0/22 #ServerIUS 31.220.40.0 - 31.220.45.255
31.220.44.0/23 #ServerIUS 31.220.40.0 - 31.220.45.255

Note: I didn't compress the first two /24 HostHatch into a /23 because I am using them in different rules.

BTW - I think we're supposed to post in this newer Server Farm thread: [webmasterworld.com...]

[lucy24]

I think we're supposed to post in ...

Oh, oops, I thought it was weird that the newest post in this thread was from two weeks ago. I guess they forgot to lock the old one when starting the new one.