mod_rewrite and mod_proxy altering log stats

mod_rewrite and mod_proxy logging

[Mad_Jester]

I have a pool of web servers that is split into two smaller pools (front and back). The front end servers are running a trim apache without support for any dynamic content (php, cgi, perl, etc.). The back end servers are running a full installation with all dynamic content enabled. The front pool takes requests from users, and depending on whether the request is for static or dynamic content, services the request or proxies it to one of the back end servers using mod_rewrite and mod_proxy. The back end servers run the script and send the content back to the front and the user then gets their page.

Here is the problem. When a request comes in for a dynamic file (.php). The request is logged only on the back end server. Since the request is being proxied via the front end server, the log shows the front end server's IP address, and not the client's IP. This is a major problem for our log analysis tools (awstats currently), as we are losing data on all of those pages.

Can someone help me find a way to properly record these requests for dynamic pages to my logs, including the client that requested them, so that our log analysis tools show accurate data?

It would be best if I could just find a way to log all requests as if they weren't being redirected, to the front end server.

Here are the pertinent bits from my vhosts.conf:

RewriteEngine On
RewriteCond %{REQUEST_URI}!.*\.(ico¦jpg¦jpeg¦gif¦pdf¦png¦zip¦tgz¦gz¦bmp¦htm¦html)$
RewriteMap server rnd:/etc/apache2/conf/backend-servers.map
RewriteRule ^/(.*) [${server:web}.domain.com...] [P]

[sitz]

I am unable to reproduce this problem with Apache 1 or Apache 2; in both cases, the request to the front-end webserver is logged. In addition to my test box, I also run mod_proxy in production, and that *also* logs accesses to requests which get proxied to remote hosts to the access log. Can you post the relevant sections of your httpd.conf?

As an aside, it should be noted that you *could* use the LogFormat directive to construct a format which would log the request of the actual client's IP on the back-end hosts. LogFormat supports the logging of arbitrary headers via the %{HEADER}i option, and mod_proxy sets the 'X-Forwarded-For' header when making an outbound request; therefore, this:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

...could be replaced with this:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %{X-Forwarded-For}i" combined

...which would tack the client's IP on the end of the log line. One could even do:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

...although in that case you'd lose the IP of the front-end machine making the request, and having that /can/ be handy for troubleshooting issues later. One obvious solution to this is simply to invert the first example:
code]
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %h" combined
[/code]
...which would result in most stats packages using the actual client's IP for statistics generation, but would still give you the front-end webserver's IP for troubleshooting.

[claus]

Try investigating Apaches mod_env (and mod_setenvif), it's a very useful module. It allows you to set a custom environment variable on any incoming request (eg. "ORIG_IP:127.0.0.1") and pass that one on to the receiving back end web server.

Here, it can be captured by a rewrite condition such as this:

RewriteCond %{ENV:ORIG_IP} (.*)

And - on the receiving end you can then use the same module to reset your front end server IP address back to the original one. It might take a bit of work to find the right formula, but it can be done.

There's a few examples (unrelated to this issue) here: [webmasterworld.com...] (posts #61,62) - as well as links to the relevant Apache docs.

[Mad_Jester]

Thank you both, those are both excellent suggestions that give me a good starting point. I will post my progress. Thanks again for the expert advice.

[Mad_Jester]

After the posters here were unable to duplicate this problem I went back and had another look at my systems. Apparently the mistake was all my own. It would appear that the servers have been logging correctly and it was my oversight. Thanks for all the suggestions. Though it didn't turn out to be a problem with these modules, this did point me in the right direction.

[claus]

No problem, glad you got it solved. And, forgot to say:

Welcome to WebmasterWorld Mad_Jester :)