Redirecting if downstream site is down

how to redirect Apache traffic

[bsavard]

Hi! This looks like a very lively and informative forum.

I'm using Apache 2.0.53 and Tomcat 5.5.7. The current setup is that Apache is serving static content and dynamic content requests are sent to Tomcat using mod_jk. There are two links (signup and login) on the static pages, and these links get processed by Tomcat.
Since Tomcat needs to go down for maintenance now and then, I'd like a way to tell Apache that during that period, show an 'undergoing maintenance' page for any requests for either the login or signup pages. I'm not sure how to set that up so that it's configurable by setting a param or something instead of Apache redirecting only if it does x number of retries to reach a Tomcat instance (causes a performance issue..)

Any pointers are very much appreciated!
Thanks!

[sitz]

I'm not aware of a way to have apache health-check the back-end system and return a maintenance page, but you could do something like this:

RewriteEngine on
RewriteCond %{REQUEST_URI} ^/(signup¦login)$
RewriteCond /var/www/conf/MAINTENANCE -f
RewriteRule ^/(.*) /path/to/maintenancepage.html [L]

This checks to make sure the request is for the signup or login pages (adjust the regex as needed); if the request *is* for one of those pages *and* a file called /var/www/conf/MAINTENANCE exists (adjust path accordingly; it doesn't need to be in the apache document root), then the request will be transparently rewritten to a maintenance page. If you have PHP installed (or want to implement a CGI script), you could tweak the RewriteRule to be something like:

RewriteRule ^/(.*) /path/to/maintenancepage.php?src=$1 [L]

You could then have the script display different content, depending on which file was originally requested. If you *do* do this, then the PHP (or CGI) script *will* need to be in the Apache documentroot.

Finally, you could tweak the rewriterule so that it returns a redirect to the maintenance page instead doing it all behind the scenes:

RewriteRule ^/(.*) /path/to/maintenancepage.html [L,R]

Do *not* use the 301 redirect (R=301), as that is a 'permmanent' redirect; this is, by definition, a temporary thing, so a 302 (the default return code for the 'R' flag) will be fine.

Does this help?

[bsavard]

Hi Sitz, thanks for your input. That gives me some ideas. What I'd really like to be able to do is somehow dynamically turn on/off your rewrite code via some management interface. Ideally, it would happen like this:
1) via some mgmnt interface, tell Apache to start redirecting requests for signup/login URLs.
2) wait till existing Tomcat sessions expire so no active users get booted.
3) take down Tomcat and do maintenance.
4) start Tomcat instance.
5) reset Apache flag telling it to pass signup/login requests to Tomcat again.

I guess the real key is #1: being able to tell Apache to do the rewriting without having to recycle Apache (since that would kill existing user sessions).

Thanks for your time/input.

[sitz]

I'm not aware of an administrative interface, but the nice thing about the solution above is that all you have to do is create a file called MAINTENANCE in the proper directory. All requests for those pages will automatically start getting rewritten immediately, without the need for a bounce. As long as none of the existing users need to hit the signup or login pages, their existing sessions should work fine. When the maintenance is over, simply remove the file and normal processing will resume.

[bsavard]

Hi again. That's true. I guess I missed the point about the flag being the presence of a file. I could always create a cgi (accessible only to admins) to create/delete the file, and that could be done via a browser. That would be an effective was to do this. The only thing to consider is the performance overhead of checking for a file on the filesystem for each request - unless the result of the check somehow gets cached in memory.

An offshoot of your idea would be using a cgi to set some kind of memory-resident value which the rewrite code would check instead of a filesystem check. That would be much better performance wise. Do you know of any type of in-memory variables/flags/etc that Apache uses and that would be accessible by your rewrite code?

Thanks for your time/input!

[sitz]

Caching the stat() call wouldn't do much good, since we *need* to check for it every time; otherwise how would we know when it was gone? =) You *could* write code that cached the time of the last stat() call, and simply compared that time to the current time and only ran the stat() if, say, 5 minutes at passed (I've had to do that before), but you're not going to accomplish that by *just* using mod_rewrite.

Now that I think about it, this recipe would be a bit better:

RewriteEngine on
RewriteCond /var/www/conf/MAINTENANCE -f
RewriteRule ^/(signup¦login)$ /path/to/maintenancepage.html [L]

This way, the 'MAINTENANCE' page only gets checked for if the request is for signup or login. In answer you your other question, there are plenty of variables that are accessible via mod_rewrite, but none of them are persistent across requests (since HTTP itself it not).

You *could* do this with a CGI, however the performance penalty for running such a script would likely far outstrip the penalty of the the single stat() call that mod_rewrite would make to check for the existence of the MAINTENANCE file; if your CGI didn't make use of an embedded interpreter of some kind, I can pretty much guarantee you that penalty incurred by a single stat() of a local file (and *do* be sure that the file is on the local filesystem) will be nothing compared to the fork()/exec()/compile-script-to-bytecode penalty you'd get by making use of an external interpreter.

To test this, I ran a perl script to make a stat call, timed the stat call in the script and the full execution time from outside the script. The script run time was .023 seconds. The time to execute the stat() called was .014 *milliseconds*. That answers that. =)

[bsavard]

I like that it now only checks the fs when either of the two URLs are requested - that'll improve perf.
Regarding setting a var/flag - I was thinking of setting it in Apache's mem, not in an HTTP session. That would make more sense because Apache should not check a session for the flag, it should check it's own internal mem space for the flag.
Checking the fs on each signup request won't be too bad unless lots of people are signing up - but that's the point of the site :) Same goes for the login link, but I think this is all doable.

Now that I think about it, all inbound traffic will go through a firewall - maybe I should explore configuring it as a type of layer 7 switch..? Maybe iptables can do that, though it's probably a bad idea to mix security with request rerouting..

Thanks again.

[sitz]

Regarding setting a var/flag - I was thinking of setting it in Apache's mem, not in an HTTP session. That would make more sense because Apache should not check a session for the flag, it should check it's own internal mem space for the flag.

*nod* There /are/ webservers out there that allow one to do this sort of thing, but Apache (vanilla Apache, anyway) isn't set up for this sort of thing. One could implement such a feature as a module (or in perl, if one's using mod_perl), but the rewrite solution is going to be less overhead than either of those, by a long shot. =)

Implementing something like this *could* be done in IPtables (I think; my iptables-fu is only medium-strength); the fact that you only want the maintenance page to appear for requests for two URLs means that IPTables (or firewall) will need to check the contents of incoming HTTP packets; this may or may not be a performance improvement over the mod_rewrite solution. In my mind, this is a job for the application, since we're dealing with things in layer-7, but I'm sure there are networking devices that could do this.

One final note; if the signup/login pages are HTTPS, the network-based solution probably won't work at all, since the firewall/network layer would have no way to decrypt the incoming HTTPS traffic. There *are* exceptions to this rule; some network devices terminate the HTTPS connection (the idea being they can open an HTTP connection to the back-end realserver and send the data in the clear, effectively offloading the expensive SSL operations from the webserver), but in order to do this, the device needs the SSL certificate in use by your site. You may or may not be set up for this sort of thing.