Forum Moderators: open
How to benchmark a CMS? How do we compare -- as objectively as possible -- system A with system B?
What metrics sd be persuasive?
I know the whole "which CMS us best" debate is endless and tends to be rather subjective -- so I am wondering if we can side-step the subjective stuff and try to find some concrete metrics.
I am familiar with CMSMatrix, unfortunately they mostly compare feature set. The qualitative aspects of their testing matrix are dictated largely by site visitor votes and as a result it has more attributes of a popularity contest -- with stronger communities pushing more votes than smaller communities, etc. That is the sort of thing I am trying to avoid.
Don't get me wrong -- I do see clear value in CMS Matrix, but it's not the sort of thing I can slap down in front of a CIO and say "System A outperforms System B in the following areas."
What sort of qualitative aspects of the CMS packages were you considering?
One of the key difficulties is how to set up a level playing field. I think it likely we will all have to accept variations in configurations, but you cd certainly test the base (default packages) against each other.
I guess the question would need to be what sort of indicators would impress said CIO enough to choose one platform over another. I would think that might end up being a lot of the list of features on a CMSMatrix type of site. However, I'd be interested to hear what others think.
To respond to Bill's last comment, you may want to read the chapter in the JMeter manual entitled Help! My boss wants me to load test our web app! [jakarta.apache.org]
Anyway, there are some good articles I've seen on benchmarking server load various CMS in shootouts. I think Dries Buytaert (drupal founder) benchmarked Drupal and Joomla, with and without various cache settings. I've seen some other tests. I would figure out your target platforms and then start doing searchs on "cms1 cms2 benchmarking" and see how they did it.
You may or may not be able to use their data. Wordpress is supposedly getting faster with each release. There's some debate WRT Drupal and in part it depends on how aggressive your caching is. So many variables!
Just recently, Crooks and Liars benchmarked Wordpress versus Drupal and switched to Drupal because of load, although Matt Mullenweg says that he doesn't think scaling is a reason to switch CMS, basically saying better caching is the answer, not a different platform (full discussion here [drupal.org]).
So how to do it?
You can use ab (Apache Bench [httpd.apache.org]), JMeter, or Microsoft WAS (Web Application Stress) to simulate a given load and see how things work. That's the easy part.
Where it gets tricky is in the details
- Are you comparing both systems under the caching scenarios you would likely use? If not, the test is worthless, because some dogs become cheetahs with proper caching and some middle of the pack CMS stay middle of the pack. So you have to make sure that's set up right. Some CMS cache differently for anonymous and authenticated users. Will you be able to test that? Does it matter in your scenario?
- Not all operations are equal. So you would likely want to figure out what your bottleneck is and test for that. For example, some CMS serve up existing pages super fast, but are very slow at adding comments. So if you have a very active community, the comparatively small number of comments getting added would be the important thing to test, not the higher page views. JMeter lets you test GET and POST requests, so it might let you test adding comments (I'm not sure about that, but see section 5.4 of the JMeter manual [jakarta.apache.org]
- you need to populate each tested CMS with realistic data and all the plugins/modules you'll use, or at least the ones that will contribute to slowing things down.
Once you've benchmarked the whole app, there's also the issue of benchmarking, or I should say performance profiling, the application you decide on. So you do all your caching, you get your server tuned, but then you can go hunting for bottlenecks in the application.
For this, on a PHP app, I use XDebug profiling and then use CacheGrind or WinCacheGrind to analyze the data. That will tell you where the script is spending its time (i.e. what functions). I had a drupal site that was slow and timing out and when I tested it, it turned out that the big bottleneck was a call to the native PHP function getImageSize(), which I was using multiple times (only six I think) for a custom block on the front page. I simply quit getting the image size and didn't have widht and height in the img tag and this sped things up a lot. I would *never* have thought that could be causing the holdup and that still makes no sense to me, but there it was in the data and changing that fixed the problem.
