Where to Place WP Related htaccess Directives

/home/useraccount htaccess OR home/usr/public_html

[Webwork]

It's my (mis)understanding that htaccess directives should be placed as high as possible in the server / virtual server's directory structure. I say (mis)understanding since I may be conflating advice about putting directives, IF possible, in a server's httpd.config file - so the directives "load once" for all virtual hosts and/or the directives are run before http traffic even arrives are a given virtual / website.

Sooooo . . I'm thinking that it "may" be possible to block certain traffic before Wordpress even has to process anything.

Each virtual host (website) on my server has 2 htaccess files. One at /home/usr)htaccess and the other at /home/usr/public_html/htaccess.

The specific directives I'm referencing are these:

# HTACESS DEFENSE
<FilesMatch ^\.ht> 
Require all denied
</FilesMatch>
# WP CONFIG and XMLRPC DEFENSE
<FilesMatch ^(wp-config|xmlrpc)\.php> 
Require all denied
</FilesMatch>

Please disabuse me of any misunderstanding about placing directives "above" or "outside" the public_html directory's htaccess file. Good or bad idea? Work the way I think or not?

Thx.

[lucy24]

"above" or "outside" the public_html directory's htaccess file

The key thing to remember is that requests have to move through the whole (physical) directory structure; they don't jump straight to the directory containing the requested domain. So that leaves a lot of possible locations for an htaccess file.

Anything involving mod_rewrite almost has to be in the directory where that domain's root lives. A few specific directives also have to be in domain-specific htaccess files. (Exact details probably depend on Apache versions. Things like Options don't work on my sites unless they're in the domain-specific htaccess file.)

Have you got one directory that contains all the various sites, like
/full-path-here/example.com/
/full-path-here/example.org/
/full-path-here/example.net/
et cetera? If so, you can absolutely have an htaccess file in /full-path-here/ that's shared by all domains ... except why would you need to, if you can put the same material in a <Directory> section in the config file?

Here's how I do it. This is on shared hosting with the "userspace" structure, meaning that I don't have "primary" and "addon" domains; they're all parallel.

At the top level-- that is, the highest level I have access to-- there's an htaccess file that's primarily concerned with access control. It uses mostly mod_setenvif and mod_authwhatsit (whatever it's called in Apache 2.2) culminating in piles of "Deny from" directives that are shared by all sites. I've also got a clutch of <Files> and <FilesMatch> envelopes that are common to all sites, such as most obviously

<Files "robots.txt">
Order Deny,Allow
Allow from all
</Files>

or

<FilesMatch "\.(js|txt|xml)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

Then there's a second htaccess file in each site's own root directory. This is primarily RewriteRules but also some miscellaneous other stuff:

Options -Indexes +Includes

(same for all sites, but it didn't work at the higher level)

SSIErrorMsg "<!-- SSI error -->"

(don't remember if I even tried this on the higher level, but it made intuitive sense to put it right after the +Includes statement)

AddType text/html .html
AddOutputFilter INCLUDES .html .php
AddOutputFilterByType DEFLATE text/css text/javascript
AddCharset UTF-8 .html .php .js .css

(Heh, I seem to say that last thing twice-- in both htaccess files-- but I think it works in both places.)

ExpiresActive On
ExpiresDefault blahblah

(Again, site-specific stuff for various types of files: for example, the test site has instant expiration so I never need to refresh/reload after making changes.)

ErrorDocument 403 blahblah

(This overrides the host's built-in ErrorDocument directives-- but by using the same filename while changing the error documents' location, I can still use the host's built-in "Allow from all" directive that lets everyone get the 403 page. I haven't seen it, of course, but I have to assume it's in the config file.)

<Files "banner-icon.png">
Order Deny,Allow
Allow from all
</Files>

(This file is only used by one site, but I want all humans to be able to see it even if it's invoked by the 403 page.)

Reminder: You can have <Files> and <FilesMatch> in htaccess -- also <If> in 2.4 -- but not Directory, DirectoryMatch or Location, LocationMatch.

[Webwork]

Thx, as always, Lucy24.

[ergophobe]

Webwork,

Just to add a couple of things to what Lucy24 said on specific questions you ask.

>>htaccess directives should be placed as high as possible in the server / virtual server's directory structure.... in a server's httpd.config file

Not exactly. The big performance hit with .htaccess is not so much that it has to be read dynamically, but as Lucy24 said, every directory in the path has to be read to verify that there are no rules overriding the local settings for that directory.

So if you have example.com/some/deep/dir/structure/to/get/to/the/flle/image.jpg, the server has check 10 directories to find out whether or not there's a rule in a .htaccess file that applies in this case. Now here's the important part, this is true even if you rules are in httpd.conf.

It is the mere fact of setting

AllowOverride on

means that the server has to assume that there could be a .htaccess file in any directory in the path and check every single one of them.

So this has two implications
1. merely putting your rules in httpd.conf only gives you a minor boost. It's turning off the ability to put rules in .htaccess that gives you the bigger boost.
2. Shorter paths mean fewer attempts to find a .htaccess file. In WP this might come into play with some permalink schemes and some file storage (upload) schemes.

>>Sooooo . . I'm thinking that it "may" be possible to block certain traffic before Wordpress even has to process anything.

Absolutely. Anything you block before the rewrite to index.php will not hit Wordpress.

>>Please disabuse me of any misunderstanding about placing directives "above" or "outside" the public_html directory'

It depends a bit on how you have your server set up. I would say that a "normal" setup would be as follows
- your access rules should default to "deny from all" AllowOverride none
- then you allow access and, if wanting to use .htaccess, set web root to AllowOverride on

If you're set up this way, there is no point in having a .htacess outside of web root because the AllowOverride None tells the server that .htaccess is not allowed anywhere but within web root. But you don't need it outside of web root, because you already put your access restrictions in as defaults.

This achieves two things
1. It's a whitelist approach rather than a blacklist approach - public_html is the only allowed dir. In your proposal, you have a .htaccess one level above web root, but what about the level above that? Doesn't really make sense

2. The performance issue. You don't want the server looking for .htaccess files all the way up to server root.


So in short
- set your defaults in your main .conf file
- make those exceptions needed for public_html

This would offer both better security and better performance I believe.

[ergophobe]

PS

.htaccess files should be used in a case where the content providers need to make configuration changes to the server on a per-directory basis, but do not have root access on the server system.

[httpd.apache.org...]

There are two main reasons to avoid the use of .htaccess files.

The first of these is performance. When AllowOverride is set to allow the use of .htaccess files, httpd will look in every directory for .htaccess files. Thus, permitting .htaccess files causes a performance hit, whether or not you actually even use them! ....

And so, for each file access out of that directory, there are 4 additional file-system accesses, even if none of those files are present. (Note that this would only be the case if .htaccess files were enabled for /, which is not usually the case.)....

The second consideration is one of security. You are permitting users to modify server configuration, which may result in changes over which you have no control.

- ibid.

[lucy24]

A final detail: One of the rare differences between htaccess as such and the identical rule located in the config file-- whether loose in config or in a <Directory> section-- is the handling of regular expressions. In the config file they're compiled once, at server startup, and then the server remembers them forever. In htaccess they're recompiled every single time; in effect, the server behaves as if it has never seen this htaccess file before. Which, in some sense, it really hasn't. So in addition to the act of looking for an htaccess file, the server sometimes has to do more work processing its contents than if those same contents were located in the config file.

Now, you could hypothetically have a DirectoryMatch
:: quick detour to Apache docs to make sure I'm not talking through my hat, with corollary shudder at Apache's beloved formulation ".+/" ::
that turns off overrides for more deeply nested directories. So you have

<DirectoryMatch /blahblah/(onesite|othersite)>
AllowOverride blahblah
</DirectoryMatch>

immediately followed (in config) by

<DirectoryMatch /blahblah/(onesite|othersite)/[^/]+>
AllowOverride none
</DirectoryMatch>

so the server only has to check one place. But it's only worth it if each of your sites contains many deeply nested directories, none of which would ever need an htaccess of its own-- for example, to allow auto-indexing in one directory, or to set a different expiration time or custom 404 page.

Much, of course, depends on whether everything on the server is your own sites, or whether you've got other people's sites there as well. Better to let those other people muck about with htaccess, at the cost of a tiny performance hit, than to either give someone else access to the config file or make yourself available for any and all changes. (Don't know about other people, but I tweak my shared htaccess every week or so. Site-specific htaccess less often.)

[ergophobe]

>>the server remembers them forever

Wow! That's a long time ;-)

But it's only worth it if each of your sites contains many deeply nested directories, none of which would ever need an htaccess of its own

And this reminds me of another perhaps not final detail.

Several WP plugins expect a .htaccess in a data directory. For example, DB backup scripts. They aren't smart enough to check actual permissions on the directory, so they check for the presence of a .htaccess and, not finding such, will give you a warning about a potential security issue.

If you know your setup is correct in the .conf file, then you can ignore these, but if it were, say, client work, you would probably need to make this look right by adding the .htaccess

Though come to think of it, you could still have AllowOverride none and most of these scripts would just find the .htaccess and "validate" your security.

[lucy24]

I think we've scared him away ;)

[ergophobe]

I was just thinking the same thing :-)

[Webwork]

Nah. Spent the day . . wasted the day . . attempting to get Verizon to NOT terminate my DSL/Phone for a $1,000. "overdue bill" that is neither "over" nor "due". After 7 months of various people promising to right this wrong, today another 2 or 3 people promised to fix the issue . . . and then . . wait for it . . Verizon still suspended my service . . and refused to reinstate it unless I paid a certain amount of ransom. For the sheer hell of it, I ended my day by contacting (form submit) Verizon's "General Counsel" and 1) asked him to examine my (absurd) Vz account ; and, 2) asked for an address for me to serve my lawsuit papers.

So, me, scared . . after a day of battling a sadistic corporate behemoth? Scared by a little regex and the arcane logic of servers?

Umm, yeah, I'm frickin' terrified.

[Webwork]

For the record: 1) It's my VPS with only my sites hosted on it, all of which run WP; 2) I'm in disbelief (believe but stunned) that I can load the standard/traditional WP rewrite rules, needed for "pretty permalinks", in the httpd.config file x1 and get all the WP sites to play nicely; 3) I'm mind boggled about how to implement this (below) in the .config file and would be happy to pay someone who knows where the landmines are before I blow everything to hades.

My current "needs" as I see them (in my 2 existing htaccess files for each WP site):

In /home/user/htaccess:

<IfModule mod_deflate.c>
AddOutPutFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/javascript application/rss+xml application/xml application/json image/x-icon
</IfModule>

# BLOCK BLANK USER AGENTS
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule ^ - [F]

# HTACESS DEFENSE
<FilesMatch ^\.ht>
Require all denied
</FilesMatch>

# WP CONFIG and XMLRPC DEFENSE
<FilesMatch ^(wp-config|xmlrpc)\.php>
Require all denied
</FilesMatch>

# Expires Defaults
<ifmodule mod_expires.c="">
ExpiresActive On
# Set default expires to 2 days
ExpiresDefault A172800
ExpiresByType text/css A31536000
ExpiresByType application/x-javascript A31536000
ExpiresByType text/x-component A31536000
ExpiresByType text/html A3600
ExpiresByType text/richtext A3600
ExpiresByType text/plain A3600
ExpiresByType text/xsd A3600
ExpiresByType text/xsl A3600
ExpiresByType text/xml A3600
ExpiresByType image/bmp A31536000
ExpiresByType image/gif A31536000
ExpiresByType application/x-gzip A31536000
ExpiresByType image/x-icon A31536000
ExpiresByType image/jpeg A31536000
ExpiresByType application/pdf A31536000
ExpiresByType image/png A31536000
ExpiresByType image/svg+xml A31536000
ExpiresByType application/x-font-ttf A31536000
ExpiresByType application/zip A31536000
</ifmodule>

# HEADER CACHING
<FilesMatch "\.(gif|jpg|jpeg|png|ico|html|css|txt|xml|js)$">
Header set Cache-Control "max-age=2592000"
</FilesMatch>

# CUSTOM ERROR DOCUMENTS
ErrorDocument 400 /errors/badrequest.html
ErrorDocument 401 /errors/authreqd.html
ErrorDocument 403 /errors/forbid.html
ErrorDocument 404 /errors/notfound.html
ErrorDocument 500 /errors/server.html

AND in /public_html/htaccess (for pretty permalinks):

# BEGIN WordPress
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
# END WordPress

[lucy24]

2 existing htaccess files for each WP site

Now, that's definitely one more than you need. Since WP is built around mod_rewrite, you may as well keep one htaccess file per site-- the one containing the actual WordPress stuff. The rest can go in <Directory> sections of config.

RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule ^ - [F]

Sure, but if you use mod_rewrite you'll have to repeat the same rule in each separate htaccess file. You can achieve the identical result with

BrowserMatch ^-?$ keep_out
Deny from env=keep_out

replacing "Deny from" with whatever locution 2.4 prefers. I kinda think the simplest way, if you're blacklisting rather than whitelisting, is to collect all the things you don't want, and shove them inside a <RequireNone> envelope, which can go anywhere. So

Require env keep_out

only don't quote me, because I don't have any personal experience with 2.4.

:: detour to docs, followed by hasty retreat as I remember the last time I tried to decipher the "expr space blahblah" business ::

The ErrorDocument directives can probably also go in a <Directory> section of the config file. This is not the kind of thing that will change very often, so no point in making the server read it all over again on every request.

[Webwork]

Thx Lucy, very much so once again. I'll keep plugging away.

[ergophobe]

BTW, lucy24 keeps throwing out variations of this

...whatever locution 2.4 prefers...

And looking at your directives, it looks like you know that she's in earnest and you have already read this
[httpd.apache.org...]

And I'm just going to say that the notes in that document are most definitely not "suggestions" (speaking as someone who crashed all sites on a VPS when I updated to 2.4 without reading said doc carefully enough).

You already have

Require all denied

So that's good. The one other thing to keep in mind is relevant to this discussion is

-- AllowOverride defaults to None in 2.4 --

This means you have to assume that .htaccess files will not work in any directory where they have not specifically been turned on (so this goes back to me comments about .htaccess above web root).

[Webwork]

Thx, ergophobe.

(Un)Wise as I am, before starting this dialogue (and after reading about the 2.4 upgrade default to "None") I asked my VPS host's NOC to either set AllowOverride to "All" or confirm that "All" was the existing setting (thinking there might be an "inheritance" from prior settings OR that their (Servint's) system for managing their server's might automagically force a setting to "All" due to the many WP users who won't tough httpd.config. The confirmed that it was already set to "All" but never answered my follow up question about "was that setting inherited or was it imposed by their server management software?"

Such interesting times. Thanks, very much so, for the most helpful guidance.

[shri]

Guys - Would life not be easier for new / non-server admin types to use something like CloudFlare for WAF / firewall / IP restrictions? Their free and pro tiers are very reasonable and do a lot of heavy lifting that shared / VPS servers are just not able to.

(Sorry! Delete if this is off topic ... but even though I've been admining nginx / apache for well over 15 years now, I think much of what these server rules do can be put on the edge)

[Webwork]

Shri, I already have created, but not implemented, a free CloudFlare account. Here's why:

CloudFlare (CF), like most startups, offers "free for now" services. That won't last.

CF's service, as I understand it, works (and will be billed) on a per site/domain basis. You know where this is heading for me. ;)

Given how the bulk of craptraffic comes from a few countries I suspect we will soon see offers of "country IP blocking servers" from the server farms as an inducement to use their services. I'd see that offer as a win-win, since so much traffic is just a waste of bandwidth and a source of server/site hacks.

One goal I have is to go as far as I can without outsourcing my (little) headaches', i.e., incurring recurring expenses.

I also wanted to better understand "the product" that I, like tens of millions of others, has allowed ourselves to become dependent upon for our livelihood.

Site speed, optimization of server services, grasping how plugin bloat and code hazard can be reduced by grasping the basics of how a plugin "does its magic" has been beneficial. For example, why install "another plugin" when a few lines of code will accomplish the same outcome? Why install a bloatware plugin when so much of the "security plugin's" defenses are more the appearance of risk than likely risk? (A 1,000 lines of code where 100 lines do 99.999% of the work.

I'm not opposed to using a CDN.

So, Shri, since you raised the topic would you care to share a bit of your experience? Do hack attempts manage to penetrate CF's "defenses"? Has CF had times when it's network / solution slowed down things a bit? Has CF been hit with a DDOS attack, ~retribution? Has using a CDN created new issues for you or for others? Such as?

[Webwork]

P.S. to Shri.

When you (or anyone else) is relying on CloudFlare or another CDN to improve content delivery (speed) OR security is there reason to not enhance security / speed at the local server level?

[ergophobe]

>>Do hack attempts manage to penetrate CF's "defenses"?

As a Cloudflare user who had a server burned during Drupageddon, I will say "yes." I will add that they can most certainly penetrate Webwork's defenses as well.

>>Has CF been hit with a DDOS attack

Of course, but it's nothing like when your server gets hit and you are trying to manage it between yourself and whatever technical expertise is on hand at your host right now.

For example, we had a customer that received an attack that was 500Gbps. This is highly uncommon and was reported as the largest DDoS in history - CloudFlare mitigated the attack for our customer.

-src: [support.cloudflare.com...]

How do you think Servint is going to do against that?

So yes, you are adding a link in the chain, but you are also adding a layer of protection. You could be burned if the link breaks, but you could be saved if the Cloudflare heatshield deploys before your server burns down.

Cloudflare will block a HUGE number of requests, a tiny portion of them legit. For low-traffic sites, 90% of the traffic might get blocked. I have a set of low to almost zero traffic sites on a free CF account that I would let a trusted friend peek at if needed. In most cases, this will speed up your response times as it reduces direct load on your server. But again, if CF is having massive attacks, it could work the other way.

Also, you are not really locked into CF in the way you are locked into hosting or a registrar which have a PITA factor to switching. If you can repoint your A Records en masse, you can turn Cloudflare off in 20 mins plus whatever the TTL is on your domains.

Even if Cloudflare just decided tomorrow to block you without notice, all you lose is the traffic until you can log into your registrar and repoint the A Record straight at your server.

I'm not saying you *should* and I don't know how that works out if you're talking about thosands of domains, but it's is something to fear less that I think you are.

The one bummer for you is the WAF component is only on paid plans, so as you say, if you want to protect 997 sites, you're looking at $5000 per month. You can mix and match pro and free sites and protect just key sites.

It most definitely gets expensive if you're running a ton of sites and they all need WAF

[lucy24]

Obligatory disclaimer for readers coming along later: At this point the thread title has become a little misleading. 99 times out of 100, when you see the words "htaccess" and "WordPress" in a subject line, we're dealing with someone on shared hosting, with only some rudimentary knowledge, making their first ventures into Apache via htaccess. webwork's situation is obviously different ;)

[Webwork]

FYI, I have configured my VPS to run mod_security + rule set(s) AND ConfigServer Security & Firewall (plugin), all of which I consider an unfortunate PITA.

@Lucy24 - In that case this thread MAY prove useful as a source of insight into questions to ask or what to look for in a shared hosting environment. I have been hump busting to understand and tweak the balancing of "security" and "speed" via htaccess and alternative paths. Fool that I am, after having invested waaaay too many hours "in that process" I am looking to share what I've learned AND to expand upon that understanding and/or knowledge . . or, in my case, the absence thereof.

[lucy24]

I have configured my VPS to run mod_security

Ooh, good for you. My host offers it as an optional extra, but it only works in the config file, so you're constrained to their rulesets. It returns a 418 error (I think the default is something 500-class but it's obviously to your advantage to set something distinctive) so it's easy to spot which lockouts are theirs. I think, however, their 418 will be overruled by any 403 originating from your own htaccess. So it's most useful for detecting unwanted visitors who would have gotten in, like when your neighbor was fortuitously walking his Rottweiler on the day you forgot to lock the door.

[shri]

Shri, since you raised the topic would you care to share a bit of your experience? Do hack attempts manage to penetrate CF's "defenses"? Has CF had times when it's network / solution slowed down things a bit? Has CF been hit with a DDOS attack, ~retribution? Has using a CDN created new issues for you or for others? Such as?

>> Do hack attempts manage to penetrate CF's "defenses"?

Start with the fact that nothing and I mean nothing will prevent an experienced and determined hacker from getting access to your interwebs. My issue is - as I look at CMS driven sites (Wordpress / Joomla / Drupal / PHP stuff) I am well aware that I need to protect them. For me, Cloudflare acts as a first (not the only) line of defense which keeps the usual trouble makers away. Their WAF (Web Application Firewall) is very decent for newbies. It will protect against the most obvious attacks that take down sites en-masse. It buys you time to protect your sites from vulnerabilities while the coders fix the problems.

Cloudflare is NOT a substitute for keeping your software updated and your sites (including SQL databases) backed up on and offsite on a regular basis.

They have mitigated some large DDOS blackmail type attacks on my ecommerce business (two weeks before xmas!) over the past so I am bit of a fan.

>> Has CF had times when it's network / solution slowed down things a bit?

Depends. A user typically connects to Cloudflare which then examines the request and sends it over to your server if it is not cached. Most pages created by CMS driven sites are not cacheable - some work has to be done to achieve this. We send headers and set rules which makes Cloudflare cache wordpress pages for a few mins. In addition to this, we clear wordpress caches through our CMS (custom & plugin code) when a page is changed or updated. To add to my specific situation, we also use varnish between cloudflare and our server. But that takes it way out of the realms of this discussion.

After all of this, the html / css / js / images are then delivered to the server. My calculations show about 50ms delay being added by Cloudflare on their basic plans. On a couple of high traffic sites we use a feature of theirs called Railgun which has even better performance ($200 / month / domain).

The performance between their free & pro plans is very similar.

Just the bandwidth savings for me pay for the bills. I am on Google cloud where every byte going out of the server is paid for. (think 100mbps traffic vs 5mbps with cloudflare as a CDN).

>> Has CF been hit with a DDOS attack, ~retribution?

Every day, every hour, every minute! I was however amazed at how well they dealt with a state sponsored attack on some democracy sites here in Hong Kong. Bottom line is that they know how to deal with this far better than any webmaster I have come across.

When we dealt with a very severe DDOS (6 gbps) we were able to mitigate a fair bit by configuring varnish / nginx / iptables on a server. The bandwidth cost was astronomical ($3xxx / day). Cloudflare took this over a few days later and mitigated it for us (total cost was $200).

I by far not a large business or even a medium sized business owner. I'm pragmatic about cost v/s downtime. I also value my blood pressure and health as I tend to get stressed and take these stupid attacks way too personally! Cloudflare for me has been cheaper than buying Zantacs and Exforge. :)

>> Has using a CDN created new issues for you or for others? Such as?

Yes - but easy to deal with. Problems we've had include:

- Country Blocking. For us, most of Africa, many parts of south east Asia and few other countries are problematic. We used to use IPtables firewalls to block all traffic from these countries. Now that we rely on Cloudflare, we've had to do a fair bit to add IP blocks into their firewall. Their blocks are smaller than what we could do - so sometimes large ISP (Afrinic for examples) which could be blocked by a single CIDR needs to be broken down into dozens of rules. Again, mostly one time stuff which protects all our sites.

- CMS issues. Many CMS like Wordpress use IP addresses for various things. We've generally had to patch them (or use plugins - which I abhor as they tend not to be well maintained) to detect that the traffic is coming from Cloudflare and to reset the IP variables.

- Cloudflare connection issues. Sometimes - every few months, for several hours there might be problems from some areas. We in particular have a problem every 3-6 months from Hong Kong - again no big deal, I'm paying $5 for the service!

Now that I'm done with my fan boy testimonial let me explain how my "stack" works.

DNS Servers - This bit is uber important for me. I set my sites up using DNSMadeEasy. They're tested, all the TXT, MX, CNAME addresses are configured on here first. The registrar points to this for the first weeks / months of operation and development. Once we're ready and have traffic we export these DNS records and import them into Cloudflare and switch the DNS on the registrar to Cloudflare.

Keeping two sets of DNS records is tedious, but it allows me to switch easily for situations where "crap happens".

Our cloudflare setup works in "Development Mode" for a while on the free plan. This means we use Cloudflare as a free SSL / DNS provider. No acceleration / firewall yet.

Once I'm happy that things are ok, we turn on the accleration and start enabling things like WAF, minify, add page rules etc.

Once traffic increases, I can then turn on a pro plan. If traffic (and more importantly revenue) exceeds some threshold, I'm happy to upgrade that site to their business plan and turn on Railgun.

Once fully turned on, my traffic flows like this - yes, it is extremely non-trivial and months and months of work has gone into this setup.

User -> Cloudflare -> "Server 1" which is Varnish / Nginx (running pagespeed) -> Google Loadbalancer -> GCE autoscale Nginx / PHP servers -> Google Cloud SQL and NFS + Memcache + Other caches.

So my bottom line recommendation stands

- If you're a newbie running on shared hosting / VPS and do not know how to admin servers / apache and all that stuff - consider cloudflare as a front end.

I know this has gone well beyond the initial question - apologies for the digression. :)

[lucy24]

A user typically connects to Cloudflare which then examines the request and sends it over to your server if it is not cached. Most pages created by CMS driven sites are not cacheable - some work has to be done to achieve this. We send headers and set rules which makes Cloudflare cache wordpress pages for a few mins.

I can't help thinking that any utility that forces caching of CMS page content is almost bound to offer performance benefits :) So that gets weighed into the mix.

[ergophobe]

>> caching

Cloudflare is a whole ecosystem. It's a WAF, a reverse proxy, a DNS manager, a CDN, a "pagespeed" style minifier, etc. You must use the DNS management for it to work, but pretty much everything else can be turned on and off. But as Shri says, most dynamic sites will offer limited caching unless you have some plugins and such that help with that.

[Webwork]

{{{Shri}}} <- A manly, manly big hug of thanks and appreciation for a truly informative / educational . . digression. ;)

Thank you for taking the time to speak to and illuminate the realities of high traffic sites.

Thank you all for helping me to better understand the issues / solutions.

[ergophobe]

If you're a newbie running on shared hosting / VPS and do not know how to admin servers / apache and all that stuff - consider cloudflare as a front end.

I'll add to this and my comments that Cloudflare is not the only player. I've also used Incapsula (though not for a few years; don't know current status). There are several others if you have a particular thing against Cloudflare (rather than the general concept of a reverse proxy service).

I've worked on a site that ran their own reverse proxy front end, load balancer, firewall and to some extent CDN (US and EU locations). All of this took seven servers (for just the US version, never got involved with the Dutch version) for one site and pretty much a full-time admin and I don't think the results were any better (and when they went wrong and said admin was out, sometimes much worse) than a free account on Cloudflare.