Reverse-Proxy incorrectly formatted calls to api (400 error)

[gunderwood]

Hi,

I have been trying to configure a reverse proxy for a back-end embedded IIS server. The back-end is a Power BI Report Server. The Apache server is version 2.4.43. Looking at comparable traces between a direct call that is successful and going through the reverse proxy where I am getting 400 errors there are the following differences. All the errors are api calls that are throwing the errors. Once the page hits the 400 error the BE throws an error message and stops loading.

Example 400 response: GET /reports/api/v2.0/System/ReportServerRelativeUrl HTTP/1.1" 400 - "https://..."


- “br” is added to the Accept-Encoding header
- The following headers are added:
sec-ch-ua header
sec-ch-ua-mobile header
sec-ch-ua-platform header
sec-Fetch-Site header
sec-Fetch-Mode header
sec-Fetch-Dest header

I have tried to use the "Header always unset" without success. The following modules are in the httpd.conf file.

LoadModule log_config_module ${APACHE_INSTROOT}/modules/mod_log_config.so
LoadModule status_module ${APACHE_INSTROOT}/modules/mod_status.so
LoadModule setenvif_module ${APACHE_INSTROOT}/modules/mod_setenvif.so
LoadModule version_module ${APACHE_INSTROOT}/modules/mod_version.so
LoadModule mime_module ${APACHE_INSTROOT}/modules/mod_mime.so
LoadModule unixd_module ${APACHE_INSTROOT}/modules/mod_unixd.so
LoadModule autoindex_module ${APACHE_INSTROOT}/modules/mod_autoindex.so
LoadModule alias_module ${APACHE_INSTROOT}/modules/mod_alias.so
LoadModule env_module ${APACHE_INSTROOT}/modules/mod_env.so
LoadModule socache_shmcb_module ${APACHE_INSTROOT}/modules/mod_socache_shmcb.so
LoadModule negotiation_module ${APACHE_INSTROOT}/modules/mod_negotiation.so
LoadModule include_module ${APACHE_INSTROOT}/modules/mod_include.so
LoadModule dir_module ${APACHE_INSTROOT}/modules/mod_dir.so
LoadModule headers_module ${APACHE_INSTROOT}/modules/mod_headers.so
LoadModule authz_core_module ${APACHE_INSTROOT}/modules/mod_authz_core.so
LoadModule authz_host_module ${APACHE_INSTROOT}/modules/mod_authz_host.so
LoadModule proxy_module ${APACHE_INSTROOT}/modules/mod_proxy.so
LoadModule proxy_http_module ${APACHE_INSTROOT}/modules/mod_proxy_http.so
LoadModule proxy_balancer_module ${APACHE_INSTROOT}/modules/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module ${APACHE_INSTROOT}/modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bybusyness_module ${APACHE_INSTROOT}/modules/mod_lbmethod_bybusyness.so
LoadModule slotmem_shm_module ${APACHE_INSTROOT}/modules/mod_slotmem_shm.so
LoadModule filter_module ${APACHE_INSTROOT}/modules/mod_filter.so

Any help is appreciated!

[not2easy]

Hi gunderwood and welcome to WebmasterWorld [webmasterworld.com]

A 400 response indicates a syntax error in the request. I am not the person who can assist with this error but I wanted to welcome you and let you know there is hope.

[gunderwood]

Thanks for the welcome not2easy :-) I have had our Apache vendor (operations only) as well as Microsoft look at this issue with me over the last 3 months with no success so far. I am hoping someone in the forum has seen this before. All the google search result recommendations have been implemented with no success.

[phranque]

welcome to WebmasterWorld, gunderwood!

All the errors are api calls that are throwing the errors.

what do these errors look like?
any hints there?

- “br” is added to the Accept-Encoding header
- The following headers are added:

where are you seeing this?
the back-end server?

any chance the error is cookie-related?

[gunderwood]

The Security part of the header - Anyone know which Apache module would be manipulating he security part of the header?

Security
Authorization: NTLM TlRMTVNT...
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin

Thanks

[robzilla]

- “br” is added to the Accept-Encoding header
- The following headers are added:
sec-ch-ua header
sec-ch-ua-mobile header
sec-ch-ua-platform header
sec-Fetch-Site header
sec-Fetch-Mode header
sec-Fetch-Dest header

By Chrome, presumably? It does that for all requests. I doubt they could be a problem.

What does your proxy config look like?

[gunderwood]

You are right... that is what I get for listening to Microsoft support -__-

The standard config file used by our company has a large amount of header edits. I have removed all of them to see if that was the issue. The current config file is as simple as it could be. This has been very frustrating to say the least.


<?xml version="1.0" encoding="UTF-8"?>
<pr:projectconfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:pr="http://com.apache.httpd.staging/ProjectConfigSchema"
xsi:schemaLocation="http://com.apache.httpd.staging/ProjectConfigSchema/ProjectConfigSchema.xsd">



<serverconfig>
<includeconfig>
<include filename="${APACHE_PROJ}/vhost.conf"/>
</includeconfig>

<logconfig>
<loglevel severity="debug"></loglevel>
</logconfig>


<proxyconfig>
<proxyrequests>off</proxyrequests>
<proxypreservehost>on</proxypreservehost>
<proxypassblock>
<proxypass url="https://example.net/" path="/"/>
</proxypassblock>
<proxypassreverseblock>
<proxypassreverse url="https://example.net/" path="/"/>
</proxypassreverseblock>

</proxyconfig>
<sslproxyengine>on</sslproxyengine>

<sslconfig>
<sslproxycheckpeercn>off</sslproxycheckpeercn>
<sslproxycheckpeername>off</sslproxycheckpeername>
</sslconfig>

</serverconfig>


vhost.conf file:

<VirtualHost *:7874>
ServerName example-rp.net
SSLEngine on
ProxyPreserveHost On
ProxyPass / https://example.net/
ProxyPassReverse / https://example.net/
</VirtualHost>

[edited by: not2easy at 7:33 pm (utc) on Mar 4, 2022]
[edit reason] exeplified for privacy/security [/edit]

[not2easy]

Pardon my edits, done only for code readability and to keep private things private. ;)

[gunderwood]

what did I miss?

[gunderwood]

Oh... I see.. my urls were all fake :-)

[phranque]

what did I miss?

it is all explained in this thread which is pinned to the top of the Apache Web Server forum thread index:
IMPORTANT: Please Use Example.com For Domain Names in Posts [webmasterworld.com]

[robzilla]

Hmm. Can you find the proxied requests in the log files on the Power BI Report Server? It might point to a more specific reason for the 400 status code, e.g. "RequestLength" when the maximum request length is exceeded.

[gunderwood]

Hi Robzilla, The log files for Power BI are not showing any 400 responses. We were only able to determine the 400 response was coming from Power BI by using wireshark

[gunderwood]

Hi phranque, My apologies... note taken.

[robzilla]

We were only able to determine the 400 response was coming from Power BI by using wireshark

Tricky. I suppose you can't inspect the data and headers, since it's going over HTTPS?

When you look at the response headers you get back from the Power BI Report Server, using a direct request (no proxy), does anything stand out there? Any particularly long header values, perhaps. You probably already looked at that .

[gunderwood]

Hi Robzilla,

I redirected the reverse proxy to port 80 so we could see the response headers. I can see from Wireshark the get request and then the response to the request. I just noticed the failed calls are all categorized as just http in the response from the server and the successful ones are categorized as http/json.

[robzilla]

Well, yes, a failed request will result in an error page, which is generally of the content-type text/html. This is normal.

Based on the headers you posted, your report server uses the NTLM authentication protocol, which apparently does not easily lend itself to an HTTP proxied environment:

It relies on authentication (an affair which involves a handshake with a couple of initial 401 errors) and subsequent connections to be done through the exact same connection from client to server. This makes HTTP proxying nearly impossible, since each request would usually go through either a new or a random connection picked from a pool of open connections.

[stackoverflow.com...]

A few possible solutions mentioned there. There's an Apache module [github.com] but using HAproxy as a TCP proxy looks to be the simplest, if that could work for your set-up.

Not 100% sure it would explain the 400 errors, but it might. HTTP proxying is clearly problematic here.

[phranque]

I just noticed the failed calls are all categorized as just http in the response from the server and the successful ones are categorized as http/json.

could you please explain what you mean by "categorized"?

[gunderwood]

This is in the Wireshark trace.

Success Response
7325.415964<IP Address><IP Address>HTTP/JSON580HTTP/1.1 200 OK , JavaScript Object Notation (application/json)

Failed Response
382143.665391<IP Address><IP Address>HTTP181HTTP/1.1 400 Bad Request

[phranque]

okay, fyi your "category" is equivalent to the Content-Type HTTP Response header value.

[robzilla]

HTTP/JSON (in the "Protocol" column) is how Wireshark may categorize an HTTP response with the content-type application/json.

Anyway, I think it makes sense that faulty authorization would result in a 400 status code. 401 would make more sense when authorization fails, but it looks like this particular type of authorization prevents the servers from having a proper conversation.

[gunderwood]

Hi, any additional recommendations or information I could provide? There are calls with extremely long authorization NTLM strings for both successful and failed requests. I am also getting 401 responses but that isn't crashing the portal page like the 400 responses.

[gunderwood]

Hi,

Is there a way to change the header request content type in my vhost.conf file? I have already added this code to the virtual server with no change.

# Replace original Accept-Encoding header with identity request.
#
RequestHeader unset Accept-Encoding
RequestHeader set Accept-Encoding identity

Thanks again for all the help so far. I am really in over my head on this topic.

[robzilla]

The Content-Type is not the problem here, it's just the way that the web server categorizes the response it sends back to the client. If it's detected as JSON, the Content-Type will be application/json, if it's HTML that'll be text/html, for CSS it's text/css, and so forth. Wireshark denotes the response as HTTP/JSON in the Protocol column because a successful response from the report server carries the content-type application/json. Not sure why it does that, actually, because HTTP/JSON is not a protocol, it's a combination of a protocol (HTTP) and the Content-Type of the response body (application/json). Anyway, you don't get HTTP/JSON with a failed request because the response is an error page, which is of the Content-Type text/html. The failure happens before the Content-Type is determined.

As noted above, it appears you cannot put an HTTP proxy in front of a server that uses NTLM authentication. To make that work, you'll either need a custom Apache build with the NTLM module or use a TCP proxy instead.

There are calls with extremely long authorization NTLM strings for both successful and failed requests

It's possible that Apache cannot handle such large header values. By default the maximum size is 8 KB.

You can try increasing the limit by adding LimitRequestFieldSize 200000 to your <VirtualHost> configuration, see what happens.

But if NTLM authentication is the problem, I suspect it won't be so easily fixed.

[robzilla]

Maybe this is helpful: [reda-bacha.medium.com...]

[gunderwood]

You are a better searcher than I am :-)

Looking at the article I am doubtful this will pass our security team but I can try it on my POC environment. It will still require an AD authorization model so I am not sure how that will be impacted. Thanks for the suggestion!

[gunderwood]

Hi robzilla,

unfortunately we did find a similar article and made this change 5 months back. I just didn't remember as it was too long ago.

BR,
gunderwoos

[gunderwood]

Request hitting the server directly (succesful)

GET /reports/api/v2.0/System/ReportServerRelativeUrl HTTP/1.1
Host: http:example.net
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-XSRF-TOKEN: deprecated
Connection: keep-alive
Referer: http://example.net/reports/
Cookie: XSRF-NONCE=H2VzNSR2ICuD8%2FTm2G5UNflr3jbdJDy1nI3V4wb9cRg%3D; XSRF-TOKEN=deprecated
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAIYAAABWAVYBngAAAAwADABYAAAADgAOAGQAAAAUABQAcgAAAAAAAAD0AQAABYKIogoAYUoAAAAPZHj/yKzyEGofu6rVbVqLP20AYwBiAG0AdwAxAHEAdABjADQAMwA0ADAATABXADEAMAA4ADMAMAAwADUAOQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA8WScteOB/E2wGoZ8BZmUFAQEAAAAAAAB8tzMPMDPYARLeAbQlBwzGAwAAAAIABgBNAFUAQwABABIAUwBNAFUAQwAxADIAMQAyADgABAAeAGUAdQByAG8AcABlAC4AYgBtAHcALgBjAG8AcgBwAAMAMgBzAG0AdQBjADEAMgAxADIAOAAuAGUAdQByAG8AcABlAC4AYgBtAHcALgBjAG8AcgBwAAUAEABiAG0AdwAuAGMAbwByAHAABwAIAHy3Mw8wM9gBBgAEAAIAAAAIADAAMAAAAAAAAAAAAAAAACAAAMlu0x8/lCajz44SBen71ofzYbO3nhxjWjVnz0a2pzloCgAQAAAAAAAAAAAAAAAAAAAAAAAJADYASABUAFQAUAAvAHMAbQB1AGMAMQAyADEAMgA4AC4AYgBtAHcAZwByAG8AdQBwAC4AbgBlAHQAAAAAAAAAAAA=
X-RBT-SCAR: xx.xxx.x.xxx:1765913794:3000

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 132
Content-Type: application/json; odata.metadata=minimal
Server: Microsoft-HTTPAPI/2.0
X-Content-Type-Options: nosniff
Set-Cookie: XSRF-NONCE=%2BQvxt4hYtMHvvGgBKfcKECpmwBdBMIcGFFjqOYQ0NpE%3D; path=/reports; HttpOnly
Set-Cookie: XSRF-TOKEN=deprecated; path=/reports
OData-Version: 4.0
Date: Tue, 08 Mar 2022 21:04:10 GMT

{"@odata.context":"http://example.net/reports/api/v2.0/$metadata#System/ReportServerRelativeUrl","value":"/ReportServer"}



Request going through the RP (failed)

GET /reports/api/v2.0/System/ReportServerRelativeUrl HTTP/1.1
Host: example.net
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
X-XSRF-TOKEN: deprecated
Referer: https://example.net/reports/browse/
Cookie: XSRF-NONCE=e8JvypnI8yZpquqYe2vKrm5A2Ku6IniKw4cuYGOflnc%3D; XSRF-TOKEN=deprecated; displayedContent=%7B%22hidden%22%3Afalse%7D; ai_user=Tir1a|2022-03-08T20:07:31.788Z; ai_session=MZvij|1646770051787|1646771691865
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAIYAAABmAWYBngAAAAwADABYAAAADgAOAGQAAAAUABQAcgAAAAAAAAAEAgAABYKIogoAYUoAAAAPCoQxubQokOYcrcM0VEG4rG0AYwBiAG0AdwAxAHEAdABjADQAMwA0ADAATABXADEAMAA4ADMAMAAwADUAOQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOKVCv7TwQQusuN/6uhOX7AQEAAAAAAADubZX2KzPYAb46WdbpEFVqo/UT1wIABgBNAFUAQwABABIAUwBNAFUAQwAxADIAMQAyADgABAAeAGUAdQByAG8AcABlAC4AYgBtAHcALgBjAG8AcgBwAAMAMgBzAG0AdQBjADEAMgAxADIAOAAuAGUAdQByAG8AcABlAC4AYgBtAHcALgBjAG8AcgBwAAUAEABiAG0AdwAuAGMAbwByAHAABwAIAO5tlfYrM9gBBgAEAAIAAAAIADAAMAAAAAAAAAAAAAAAACAAAMlu0x8/lCajz44SBen71ofzYbO3nhxjWjVnz0a2pzloCgAQAPPTi1lA8azFRtEI5/OXC74JAEYASABUAFQAUAAvAHAAbwB3AGUAcgBiAGkALQBoAHMALQBwAG8AYwAtAHIAcAAuAGIAbQB3AGcAcgBvAHUAcAAuAG4AZQB0AAAAAADp9ARj
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: example.net

X-Forwarded-Server: example.net
Connection: Keep-Alive

HTTP/1.1 400 Bad Request
Content-Length: 0
Server: Microsoft-HTTPAPI/2.0
Date: Tue, 08 Mar 2022 20:34:51 GMT


Differences:

Good request has this added and is not in the bad request

Accept-Encoding: gzip, deflate
X-RBT-SCAR: xx.xxx.x.xxx:1765913794:3000 (xx.xxx.x.xxx is an IP address)


Bad requesthas this added

Referer: https://example.net/reports/browse/
Cookie: XSRF-NONCE=e8JvypnI8yZpquqYe2vKrm5A2Ku6IniKw4cuYGOflnc%3D; XSRF-TOKEN=deprecated; displayedContent=%7B%22hidden%22%3Afalse%7D; ai_user=Tir1a|2022-03-08T20:07:31.788Z; ai_session=MZvij|1646770051787|1646771691865
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAIYAAABmAWYBngAAAAwADABYAAAADgAOAGQAAAAUABQAcgAAAAAAAAAEAgAABYKIogoAYUoAAAAPCoQxubQokOYcrcM0VEG4rG0AYwBiAG0AdwAxAHEAdABjADQAMwA0ADAATABXADEAMAA4ADMAMAAwADUAOQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOKVCv7TwQQusuN/6uhOX7AQEAAAAAAADubZX2KzPYAb46WdbpEFVqo/UT1wIABgBNAFUAQwABABIAUwBNAFUAQwAxADIAMQAyADgABAAeAGUAdQByAG8AcABlAC4AYgBtAHcALgBjAG8AcgBwAAMAMgBzAG0AdQBjADEAMgAxADIAOAAuAGUAdQByAG8AcABlAC4AYgBtAHcALgBjAG8AcgBwAAUAEABiAG0AdwAuAGMAbwByAHAABwAIAO5tlfYrM9gBBgAEAAIAAAAIADAAMAAAAAAAAAAAAAAAACAAAMlu0x8/lCajz44SBen71ofzYbO3nhxjWjVnz0a2pzloCgAQAPPTi1lA8azFRtEI5/OXC74JAEYASABUAFQAUAAvAHAAbwB3AGUAcgBiAGkALQBoAHMALQBwAG8AYwAtAHIAcAAuAGIAbQB3AGcAcgBvAHUAcAAuAG4AZQB0AAAAAADp9ARj
X-Forwarded-For: xxx.xxx.xxx.xxx (IP Address)
X-Forwarded-Host: example.net
X-Forwarded-Server: example.net


Question: Since I have basic authentication enabled on the back-end server how can I tell Apache to use basic authentication instead of NTLM and validate the user via windows Active Directory?

[gunderwood]

Update: One of my coworkers had changed the back end to basic authentication whilst I was in another meeting so when I went to look at the file and saw it was already set I looked at the date modified and forgot that Europe date format was being displayed. I now have another issue but its not specific to the reverse-proxy.

@robzilla, I can't thank you enough for your help and knowledge with Apache. I owe you a coffee sir!

[robzilla]

Happy to help. And I don't even use Apache ;-) nginx is my web server of choice, but you'd have the same problem there.

Question: Since I have basic authentication enabled on the back-end server how can I tell Apache to use basic authentication instead of NTLM and validate the user via windows Active Directory?

Strictly speaking Apache does not "use" NTLM, it just passes data between the client and the backend server. The Authorization header (with the NTLM value) is set by the application (or remembered by the browser for the duration of the session) on the client side, and Apache merely forwards those headers to the backend. With basic auth, however, I suppose Apache would have to request the credentials from the client when a request is made, and then forward those to the backend (I've never done this myself). The Apache docs can probably help you set that up: Access and Authorization [httpd.apache.org], Access Control [httpd.apache.org].

Or are you now saying the reverse proxy currently works with basic authentication?