Turbocharged

Everyone’s been there at least once. One of your blog pages gets dugg or slashdotted. What to do?

WordPress is often maligned as a resource-hungry blog platform — one that requires a lot of resources to run in high-traffic settings. But believe it or not, it’s possible to survive traffic spikes, while still serving pages to your audience swiftly. And you won’t need big iron to do it.

How do I know it? Because this very page was subject to the Digg effect, while (simultaneously) another domain hosted in this very server was subject to simultaneous Digg/Slashdot/Reddit/The Pirate Bay/del.icio.us traffic storms. And (after the measures outlined here) it stood its ground.

Throughout this guide, I’ll make a few assumptions on your part:

1. You’re using Apache to serve your pages, and you can change your Apache configuration.
2. You’re storing the contents of your blog in a MySQL database, the configuration of which you can change.
3. You’re running Turbocharged, or you have access to install WordPress plugins.

If these don’t apply to you, don’t worry — we’ll explore (sadly, less effective) alternative approaches so you can still fend off traffic storms.

Why WordPress blogs die weathering traffic storms

Before going deep into defending your blog from high traffic, let’s discuss a bit how WordPress works.

How pages are served when WordPress serves them

Here’s the life cycle of a page served by WordPress:

1. A reader visits your blog. Thus, his web browser requests a Web page from your server.
2. Your Apache Web server receives the request. Through the URL, Apache determines that the request needs to be fed to WordPress.
3. WordPress loads and receives the request. It processes the request, and identifies which page to show.
4. Then, WordPress generates the HTML code that will get sent to your reader’s Web browser. This generation is accomplished by assembling the results of several MySQL queries into your WordPress theme.
5. Apache is fed, in real time, the response of the request, which gets sent (in chunks) to your reader’s browser.
6. Your reader’s browser receives the HTML and processes it. The browser makes further requests for the required resources (links to images, style sheets and JavaScript scripts), and immediately submits those requests back to your Apache Web server.
7. Go to step 1 for each one of the resources that your page required. If any of these new requests require WordPress to generate a page, then call WordPress again.

In general, this is the process that takes place when a page in your site is requested.

But sometimes, WordPress can’t keep up…

Steps 3 and 4 take the most time. That is to say, WordPress is CPU- and database-bound. For example, in this blog, average page generation times dwell around 0.7 seconds. Contrast that to the time it takes to process a request for a static image, page or stylesheet — which is about 0.005 seconds — and you’ll understand why it’s monumentally huge.

This page calls fifteen other static elements, so (in theory) a full page view could be served in 0.8 seconds of processing time. Even so, under most circumstances this wouldn’t be a problem. I hardly serve one page per second constantly, and even if I did, the server would weather it just fine. If I have a second to serve all elements, and I serve all of them in 0.8 seconds, then no problem, right?

That’s the theory. In practice:

• Traffic storms aren’t “average”. When your site experiences a traffic storm, you can be sure it’s going to be ten times (or perhaps even a hundred times) larger than the average traffic.
• There’s a limited amount of simultaneous Web requests you can serve per second. With Apache, this limit is bound to the MaxClients setting. When all Apache processes are busy serving pages, further requests get backlogged in the queue. Evidently, not even static elements like images and stylesheets get their chance to be served.

In summary: under a traffic surge, your Web server might be receiving ten WordPress (dynamic) requests per second, but it will be dispatching only one in that time — and the rest of the requests will be backlogged in a queue. The queue will grow and grow. Visitors will wait, and wait, only to never see the page they expected. Then they’ll head somewhere else.

I know this because it happened here. During the aforementioned traffic storm, all Apache processes in this server (MaxClients = 13, explained below) were bogged down.

On Linux, you can be certain this is the problem if, when you launch top, you see high average CPU usage times (the state named N.NN%us on the first screen lines).

…keepalives don’t help

Keepalives minimize response times for a Web page, by simply not closing the HTTP connection and accepting a second or third one in the same channel. Problem is, while the connection is idle, “the slot is taken” and it cannot be used by anyone else until the Web browser is done sending and receiving all requests.

Slow clients tend to hog Apache processes for themselves — in the default configuration, for up to 30 seconds. If each client generously serves himself out of three or four Apache processes (hint: they all do), a wave of slow clients can render your site completely unresponsive for minutes at a time, even if you have plenty of capacity to serve.

…and sometimes, it’s the machine that dies

But why don’t you up the maximum number of clients? I hear you say. Here’s why:

Let’s take Apache as an example, again. Apache is normally configured to serve up to 256 requests simultaneously. Most hosting servers don’t have enough memory to serve so many requests. Hence, Apache will continue to start additional processes to serve up the requested pages. At some point, physical memory will exhaust — and then you’re fried: the machine dies out of memory starvation.

In this scenario, it’s not just the fact that requests come faster than they’re served; now the machine is deadlocked while Apache continues to pile up incoming requests in the queue. Too many Apache processes, and your blog becomes memory-bound.

This is not a problem when serving static content — in this scenario, you’d be hard-pressed to max out your memory with the default setting. The problem, of course, is WordPress again. Each Apache process serving WordPress will balloon to 20 to 30 MB of memory. WordPress makes filling up server memory real easy.

How to tell if this is the case? Well, use top — but good luck trying to launch it, because it’ll take ages due to disk swapping. Once it’s launched and running, you’ll see very high (almost 100%) usage in the waiting for I/O state (NN.N%wa). This is almost always a symptom that the CPU is idling, waiting for data to come in and out of swap.

Fixing this problem when it happens is incredibly hard, because the machine gets so slow that remotely logging in to shut down Apache (and free memory up) can take minutes. Often, you’ll just call your hosting provider and ask them to reboot your server — only to discover that, when it’s backed up, the traffic storm hasn’t stopped and your server is deadlocked again.

First line: avoiding out of memory deaths — and rationing keepalives

Remember the traffic storm I was talking about? A slashdotting scenario from last year taught me the following tricks, so when the traffic storm came, I was prepared because I already had carried the following steps out. More than a performance trick, this is baseline tuning you must absolutely perform if you don’t want your server to die.

The first item in our checklist is configuring Apache properly, so your blog doesn’t ever become memory-bound, or Apache process-bound. You need to mind three different configuration variables:

1. The number of Apache processes your machine can handle (MaxClients). This is directly proportional to the amount of memory each Apache process takes up, and limited by the physical memory of your machine. If your machine starts dipping into swap to serve pages, you’re screwed because page generation time will multiply itself by a factor of ten or twenty.
2. The backlog queue depth (ListenBacklog). It doesn’t make sense to keep visitors waiting if your machine won’t be able to handle them. Better to refuse them straight than to keep them (and your machine) tied up waiting.
3. Keepalives. You need to kill them. Sure, this may add a few milliseconds to each page load — that’s the preferred outcome vs. locking tons of clients out in the backlog queue.

How to do it? I wrote a separate guide that deals with the problem. In the same vein, MySQL needs to be kept in check, so check this other guide.

In general, the principle is simple: if your hosting machine starts swapping, you need to scale back the number of Apache processes immediately, and keep scaling back until swapping stops completely. On this server, with 512 MB of RAM, I had to scale down to 15 Apache processes to avoid swapping. In practice, this means I can serve up to 4 Web browsers fully in parallel, which is still fairly high.

The alternative is to purchase more memory for your server, but we know how expensive that can be, right?

Second line: MySQL query caching

The second item in our checklist is speeding actual page generation. Did you know that most of the queries that WordPress makes are the same, over and over? Most of the time WordPress spends, it does so while waiting for query results in the database. Why spend extra time processing repeat queries?

It doesn’t make sense to have MySQL generate the same data sets millions of times, so you should turn MySQL query caching on. The query cache is smart, and will regenerate itself intelligently if any content changes, so no worries about malfunctions. And it’s just one switch in the configuration file, /etc/my.cnf:

[mysqld]
query-cache-type = 1
query-cache-size = 10M

WordPress doesn’t need a huge query cache size, so I have mine at 10M. Once you’ve configured these settings in the MySQL configuration file, restart your MySQL server so it picks up the configuration changes.

This switch, alone, dipped WordPress page generation times to half; it’s a fairly big win to be able to serve twice the amount of articles with a single, harmless configuration change.

Third line: advanced WordPress caching

Turbocharged includes a fantastic caching plugin for WordPress, named WP-Cache.

What it does is amazingly straightforward and effective: the first time a page is requested, the generated HTML is saved to a cache file with a timestamp. When the same page is requested again, WP-Cache checks if it’s been cached, and if it is fresh (below a configurable amount of time, defaults to 1 hour) it instantaneously feeds the cached page to the client, and completely bypasses all WordPress engine calls.

It’s dramatically effective because, while the first generation of a page could take 1 second, subsequent requests to the same page take about 50 milliseconds. In practice, this means you can multiply the amount of clients served by 20.

And WP-Cache is fairly smart as well. If someone makes a comment on an article, the cache for the article is automatically regenerated — your visitors always see fresh content. At this point, your blog becomes only network- and CPU-bound, and the only thing you can do to make your blog serve more pages is to buy a more expensive machine and more bandwidth.

But WP-Cache also makes a compromise. Suppose you have a plugin that shows the most recent comments on your blog’s front page. WP-Cache can’t detect that, so in practice the recent comments list will be stale anywhere from 1 minute to 1 hour (depending on the expiration time setting). It can’t detect statistics plugins that depend on WordPress hooks to increment counters or save page visits, so you may see skewed statistics from those plugins.

So it’s a compromise. All in all, you lose some dynamism, but you win performance.

A compromise? Why don’t we try on-demand caching for a change?

Fortunately, UNIX and Linux are very scriptable and flexible, so it’s possible to have WP-Cache enable itself automatically when load times go unacceptably slow. Here’s a set of scripts that will do the job.

The first one enables or disables WP-Cache by modifying the WordPress configuration file in real-time. In this example script, we modify the configuration of a Web site stored in /var/www/html.

The script isn’t rocket science: it just looks for define('WP_CACHE',true) in the WordPress configuration file, and then it modifies it accordingly. This means you must already have set it in the wp-config.php file.

#!/bin/bash

if [ "$1" == "on" -o "$1" == "off" ] ; then

            [ "$1" == "on" ] && sed "s/\/\/.*WP_CACHE.*/define ('WP_CACHE', true);/g" /var/www/html/wp-config.php > ~/tmp/bibibi
            [ "$1" == "off" ] && sed "s/.*WP_CACHE.*/\/\/ define ('WP_CACHE', true);/g" /var/www/html/wp-config.php > ~/tmp/bibibi
            if [ ! -f ~/tmp/bibibi ] ; then
                    echo error: could not create temporary file
                    exit 2
            fi
            chgrp apache ~/tmp/bibibi && \
            chmod 640 ~/tmp/bibibi && \
            mv ~/tmp/bibibi ~/$a/html/wp-config.php || exit $?

else echo usage: wpcache ‘<on|off>’ echo status on $a: grep WP_CACHE /var/www/html/wp-config.php exit 1 fi

The second script uses wget and Python to time how long the front page takes to load. If load time goes over 10 second, this script runs wpcache on, turns itself into a service, waits 3600 seconds (one hour) and then runs wpcache off. I run it as a cron job, every minute:

* * * * * /full/path/to/autowpcache

That way, whenever the site becomes bogged down, the server automatically turns caching on, but only temporarily — after one hour, caching is disabled so your pages become fully dynamic again. If load times go up again, the script will re-enable caching after a few moments.

Obviously, fetching your front page may affect your Web statistics slightly — if this is a concern, place a 1×1 image in your wp-content/uploads folder and fetch that instead.

#!/usr/bin/env python

import sys import commands import time import os

start = time.time() execution = commands.getoutput("wget -q http://rudd-o.com/ -O /dev/null") end = time.time() - start

if end > 10.0: commands.getoutput("/full/path/to/wpcache on") print "Load time for Apache status page went overboard" print "Detected load time:",end,"seconds" print "WP-Cache has automatically turned itself on" print "It will turn itself off in an hour"

    sys.stdout.flush()
    sys.stderr.flush()

    try:
            pid = os.fork()
            if pid > 0: sys.exit(0) # exit first parent
    except OSError, e:
            commands.getoutput("logger -p user.err Failed to fork 1 in autowpcache")
            #print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
            sys.exit(1)

    # decouple from parent environment
    os.chdir("/")
    os.setsid()
    os.umask(0)

    out_log = file('/dev/null', 'a+')
    err_log = file('/dev/null', 'a+', 0)
    dev_null = file('/dev/null', 'r')

    os.dup2(out_log.fileno(), sys.stdout.fileno())
    os.dup2(err_log.fileno(), sys.stderr.fileno())
    os.dup2(dev_null.fileno(), sys.stdin.fileno())

    # do second fork
    try:
            pid = os.fork()
            if pid > 0: sys.exit(0) # exit from second parent

    except OSError, e:
            commands.getoutput("logger -p user.err Failed to fork 2 in autowpcache")
            #print >>sys.stderr, "fork #2 failed: %d (%s)" % (e.errno, e.strerror)
            sys.exit(1)

    # we're the child now, so we guard the wpcache for an hour
    time.sleep(3600)
    commands.getoutput("/full/path/to/wpcache off")
    commands.getoutput("logger -p user.notice WP-Cache has been turned off")</pre>

Fourth line: rationalize resources

Nobody wants to do it, but sometimes you just have to. Some plugins can add unacceptably long times to page generation. Be it because of inefficient queries, of complex ones, or increased memory requirements, you may want to take a look at the plugins you have enabled and see how they affect your blog.

Themes can also be problematic. Particularly when displaying pages with long lists of comments. If you experience such a problem, you may want to switch (even if it’s just temporarily) to a theme that displays paged comments instead of all comments in a single page. Turbocharged includes Binary Blue, which has that feature.

Fifth line of defense: opcode caches

An opcode cache (also known as a PHP accelerator) can be a valuable acquisition for your site. For example, eAccelerator can offer a two or threefold increase in processing speed when serving dynamic WordPress pages.

It’s sort of involved to install one if you have to compile it (one of the staffers at my hosting company helped me through it, because there wasn’t a ready-to-install download for 64-bit servers), but it offers quite the performance gain.

Sixth line: proxy using Squid or Varnish

Is your blog still slow, even after executing all steps up to here? Then you must consider an accelerating proxy. In the end, I had to resort to one too. When that nasty traffic storm hit us, it was only a Digg/Reddit/del.icio.us trifecta, and all measures so far had finally brought the server back up.

Little did I know I’d be linked from Slashdot and The Pirate Bay a scant 30 minutes afterwards. Before that, I’d be serving 13 to 15 requests simultaneously, and the server was accessible. But once I got linked by the two big ones, 15 parallel Web server processes didn’t cut it anymore.

Then I started to think: Apache is serving both dynamic and static requests. What if I divided the responsibility between dynamic and static requests? The “static Apaches” wouldn’t balloon to 30 MB each, and as a result I could serve a much higher MaxClients setting — more simultaneous clients!

In effect, each Apache process was consuming an inordinate amount of RAM, because WordPress needs it, but at the same time, the same processes were serving static content such as images and stylesheets (wasting memory that could be put to use in more Apache processes). Upping MaxClients was the recipe to kill the machine. Keeping it down wasn’t allowing the server to serve more visitors.

I figured it would be best to shield Apache from repeated, cacheable requests (style sheets, icons, images, and other static files). Thus, I fronted Apache with Squid. That way, all static content would be served straight without touching Apache.

Squid: the traditional choice

And you can do too. To the effect, all you need to do is move your Web server to port 81, install Squid, make it run on port 80, and tell it to cache everything. Here are example stanzas from the Squid configuration file, squid.conf:

http_port 80
httpd_accel_host localhost
httpd_accel_port 81
httpd_accel_uses_host_header on
cache_access_log /var/log/squid/access.log
emulate_httpd_log on
http_access allow manager localhost
http_access deny manager
http_access allow localhost
http_access allow all

The net effect is that:

• Pages served by WordPress are dynamically served through Apache (because they’re dynamic and Squid won’t cache them). WP-Cache also cooperates with Squid, and your visitors get content as fresh as is possible with caching.
• Static objects such as style sheets and images hit Apache once, then subsequently bypass Apache. Thus, you have more Apache processes available to serve more (dynamic) WordPress pages.

15 minutes after the second traffic wave hit this site, Squid had solved the problem. In this site, activating Squid relieved Apache of about 90% of the requests it was getting before. And this step provides the most dramatic difference: before, this server was serving 7 requests per second before; after Squid it served in excess of 70, with no impact to freshness. Obviously, general server load went down too.

Varnish: the fast and flexible alternative

Right after I published this article, I started receiving suggestions (see the comments) to use Varnish instead of Squid. Truth be told, I only used Squid because it was the easy install, and I was familiar with it. From install to startup, it took me 15 minutes — if you’re a fast typist (I am) and you know what you’re doing (I think I do), you can quickly solve a traffic surge this way.

But, later, I changed to Varnish. It has three big advantages compared to Squid: it’s more lightweight, the caching engine is faster, and it doesn’t die when it runs out of disk space. Of course, there are two disadvantages: it’s not as mature (I personally suspect the caching logic is overzealous) and I had to download a package from a Web site, instead of using my distribution’s package manager.

For the record, here’s my Varnish configuration file. (/etc/varnish/default.vcl). Note that I’ve moved Apache to port 81 and put Varnish in its place on port 80:

backend default {
        set backend.host = "127.0.0.1";
        set backend.port = "81";
}

sub vcl_recv { # these apply to my hosts

            if ( req.url ~ "amyru.h18" ) { # kick a cracker out
                    error 403 "Go away cracker";
            }
            if ( req.url ~ "wp-content/uploads/(images|200.)" ) {
            # if images, prevent hotlinking for some, permit for some
            # (read this branch to understand the logic)
                    if (
                        !(req.url ~ "wp-content/uploads/images/gravatar-picture.png$") &&
                        !(req.url ~ "wp-content/uploads/images/logos")
                       ) {
                            if ( req.http.referer ~ "^http(|s)://" ) {
                                    if ( !(req.http.referer ~ "^http(|s)://(([a-z-]+\.|)rudd-o\.com|([a-z]+\.|)turbochargedcms\.com|faviconsfor\.us|images\.google(.+)|search\.live\.com|([a-z-]+\.|)hi5\.com|([a-z]+\.|)ubuntu\.ec|([a-z-]+\.|)planetalinux\.([a-z-]+))" )) {
                                            error 403 "Hotlinking is forbidden";
                                    }
                            }
                    }
            }
            # Big files do not get cached, no point in caching a 300 MB file
            if (req.url ~ "\.(mp3|ogg|avi|mpg|flac)$") { pipe; }
            if (req.request != "GET" && req.request != "HEAD") { pipe; }
            if (req.http.Expect) { pipe; }
            # If the user is logged on WP, do not cache, shunt directly to Apache
            if (req.http.Cookie ~ "wordpresspass" || req.http.Cookie ~ "wp-postpass") { pipe; }
            if (req.http.Authenticate) { pipe; }
            if (req.http.Pragma ~ "no-cache") { pipe; }

            lookup;

}

As you can clearly see, you can even front several Web servers running on different ports. Perfect when you have customers that need complete control of their Apache configuration — now you can run an Apache server under each user account — much more secure than letting multiple users access the same Apache instance.

Anyway, this is clear proof that an accelerating proxy can really help you max out your network link!

Conclusion

Through a combination of proper configuration, resource rationalization, and advanced caching, you can make your WordPress/Turbocharged blog survive heavy traffic. Now go and apply this war chest to your blog!

Tip: why don’t you try Turbocharged today? This and other common blogging tasks become much easier with it!

This entry was posted on Saturday, April 28th, 2007 at 6:07 pm and is filed under Turbocharged and WordPress tutorials. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.