Apache, Nginx, PHP, HHVM? Test it yourself! Ansible

The most typical conversation (aside from which plugins to use) in the WordPress sphere is about the speed comparisons between various web serving software like Apache or Nginx. There are tens of articles with nice charts and big claims on the Internet: Nginx is faster than Apache. HHVM is faster than PHP-FPM and so on. The question is: can we trust them? What if the technology behaves differently on our VPS? The only way to be sure is to carry out the benchmarks ourselves. It is, however, time-consuming to install and configure the whole stack. I think I found a (partial) solution.

Ansible to the rescue

Working on my Bachelor’s Thesis, I had to find a way to quickly switch between Apache with mod_php, Nginx with PHP-FPM, HHVM, etc, because I’m comparing and optimizing their performance. Since then, I’ve learned that Ansible might help me, as it can be used to automate mundane server-related tasks.

After some struggling weeks, I’ve put together a collection of Ansible playbooks that install, set up and reset everything that we need. It is enough to input IP address (or hostname) of your VPS server, run a playbook and benchmark the WordPress site.

Prerequisites

I’ve created the playbooks to work with Ubuntu 14.04. It should be possible to use any Debian-based Linux distribution, but you will need to change the apt repository versions in the roles. Have a fresh Ubuntu installation on the server because the playbooks will manage it completely: from updating apt and setting up the locales to downloading WordPress and configuring Nginx to serve it.

First steps

1. Clone the wordpress-ansible repository.
2. Edit hosts file and add/replace IP address or hostname of your testing server. Assign the servers into the group “webservers”. Ansible Inventory — read more.
3. Open the group_vars/webservers file and modify the variables to your needs. This file holds most of the variables used during executing the playbooks and roles. Change the path to the public part of your ssh key. Ansible will ask for root password when run for the first time, then copy the ssh key so it will become passwordless. If you need to generate a ssh key, follow GitHub guide. Also change the wordpress_home_url to match your server’s IP address or hostname.
4. Optional: check the roles so you roughly know what they do.

We are now ready run Ansible. Within your command line, navigate to the cloned repo’s directory and execute following command:

ansible-playbook -i hosts install-all-software.yml

It could take between 10 to 30 minutes to fully complete all the roles and tasks. Let it work and have some coffee. If you run into some errors, open a new issue or notify me in the comment section below, please.

Let’s do some benchmarking

If the previous command finished successfully, your server is almost prepared. The last step is to “activate” (run a playbook) a specific server configuration, so we can actually test the loading speeds.

However, we still don’t have a proper tool to send the testing requests to the server. In the past, I’ve used Apache Bench command line utility to do this. The downsides of ab are that you need to have a fast Internet connection (the test is run from a local computer), there are no nice charts and no history. Nowadays, I’m using loader.io (referral link), which is a load testing service with up to 10,000 concurrent requests for free.

Setting up loader.io

When you sign up at loader.io, you can add a new server for testing. You need to verify the server by placing a txt file on your server where loader.io can access it. I’ve created a simple playbook which puts the token into the WordPress directory so its path will be http://example.com/<your-token.txt>. Here’s the playbook:

---
- name: Loader.io host verification >

  hosts: webservers
  remote_user: root
  vars:
    token: 

  tasks:
  - name: Install loader.io verification token
    shell: echo {{ token }} > {{ token }}.txt
    args:
      chdir: "{{ remote_wordpress_dir }}"
      creates: "{{ remote_wordpress_dir}}/{{ token }}.txt"

Save it in the cloned repo’s directory and run it if you reset the WordPress installation.

Nginx with PHP-FPM

Let’s test Nginx with PHP-FPM, without FastCGI and WordPress database caching. Execute the following command:

ansible-playbook -i hosts nginx_php-fpm.yml

and

ansible-playbook -i hosts wordpress_basic.yml

When you navigate to your server’s IP address in your browser, you will see the WordPress with the default theme and “Hello World” post. Good. Now, go to loader.io and create a new test. I selected the “Maintain client load” as test type and put clients from 0 to 200 of duration 30 seconds. Feel free to use different settings. Also don’t forget to run the loader.io playbook which adds the token to your WordPress site, as described before.

Using my settings and a VPS with 3 CPUs and 4GB of RAM, here’s the resulting chart, concurrent clients versus average response time:

From the chart, we observe that Nginx with PHP-FPM handles 200 concurrent connections with no problems. Link to full test. If you want to read the nginx.conf or php-fpm.conf, click here and here, respectively.

I want more! WordPress advanced, Nginx and HHVM

HHVM is a beast, no doubt about it. However, to see it shine, we will load-test it thoroughly. Run these commands:

ansible-playbook -i hosts nginx_hhvm.yml
ansible-playbook -i hosts wordpress_advanced.yml

The first playbook activates and configures HHVM. The second one installs:

• a few large WordPress plugins (you can change them in the group_vars/webservers file), such as WooCommerce, Jetpack, WordPress SEO by Yoast, etc.
• a free e-commerce theme Storefront
• WooCommerce dummy data
• Awesome WP Test dummy data

This makes the WordPress site really huge, taking a lot of memory and CPU. Let’s see how HHVM handles it:

While the response times are lower than in previous test, HHVM copes with the load quite heroically. Link to full test.

What’s next?

Can you see the beauty in this? Never ever will you have to manually install and configure all the software to be able to load test it. Just run some playbooks and you’re done!

There are a few more playbooks you can try:

• Apache with mod_php
• Nginx with HHVM and FastCGI caching (full-page cache)
• WordPress with Redis for database (object) caching

Feel free to create new playbooks by combining some roles or making new ones. If you do, make a pull request so I can merge it into the repository. Thanks.

Bonus: Use htop to measure server load

Sometimes it’s not enough to see the loading time charts to be able to arrive at a correct conclusion. Observing the server load is an important factor too. You can use the htop command line utility (installed with the install-all-software.yml playbook) to measure it. As an example, take a look at Apache with mod_php versus Nginx with PHP-FPM charts and server loads:

Apache with mod_php

Nginx with PHP-FPM

As we can see, while the average response times are similar, RAM usage is whoppingly 17 times higher with Apache and mod_php than with Nginx and PHP-FPM. I know that Apache can be tweaked. On the other hand, I don’t see how you could reduce its RAM usage to be 17 times lower. No way. Don’t use Apache with mod_php.