Stressed Servers

Initial results show that even a naive, non optimized solution with off the shelf routing and server frameworks can easily serve 60k qps on 8 CPU cores with good response times.

As of writing this post, the surprising winners are http-kit and pohjavirta. These are the only servers which managed to get good response times at high percentiles. Also, you should prefer the latest JDKs.

I expect in the future to see more servers join this illustrious bunch as the ring and reitit performance PRs make their way in.

It's not a secret Clojure isn't the fastest language in the world. Immutability and high level of abstraction come at a cost.

At that case, the reader might wonder if Clojure is a good fit for applications which deal with large amounts of data and lots of traffic. We can always provision more machines, but alas, they cost money.

Since May this year, I have been poking and prodding at different web server libraries in Clojure, combined with Metosin's high performance web tech stack.

I set up a profiling environment which used wrk2 at different rates to find the performance limits of the different libraries, across different JVM versions and GC algorithms and produced corresponding flame graphs.

Finally, after nights of experiments, the results are in - while there is still room for improvement, we can go pretty darn fast with Clojure, without special tuning or weird looking code.

In this post and followups, I'll present my findings, methodology and some conclusions and future goals.

A good measure of a server's behavior is its response time to queries at different rates over a long enough period of time.

Long enough is a slightly arbitrary measure, which requires we understand how web servers fail. The entire stack is built on queues. From the operating system kernel to the executor service on the JVM. The server will fail when one of them starts to fill up. Response times will increase until the server starts throwing exceptions, such as RejectedExecutionExceptions from an ExecutorService, or OOM due to an unbounded queue.

Once we have established the server is probably stable at an operating rate, we can measure the latency for requests handling and plot it as a histogram:

 50.000%    1.10ms
 75.000%    1.49ms
 90.000%    1.87ms
 99.000%    4.34ms
 99.900%   34.27ms
 99.990%   73.54ms
 99.999%   95.61ms
100.000%  134.65ms

We are interested in three measures:

• typical response time for some operating rate
• highest acceptable operating rate

The acceptable operating rate is usually a function of:

• The "wall" for that typical rate: note how at a certain percentiles response times increase exponentially.
• Worse case response times

These considerations are relevant to a specific application and can't be judged in a vacuum. For some cases anything above 1ms is unacceptable, while for others a worst case response time of 100ms is fine.

Here you go

There is a lot of data to wade through, as the results are a product of all these options:

• server libraries: http-kit, jetty, aleph, pohjavirta, undertow
• request handling methods: middleware, interceptors
• handling synchrony: synchronous vs. asynchronous
• JVM versions: 8, 11, 15
• GC algorithms: ParallelGC, G1GC, ShenandoahGC, ZGC
• Operating rate: from 10k to 60k qps

For which you will find HDR plots and flame graphs.

This is too much to take in. The astute reader will notice there are hundreds of possible scenarios. At this stage let us skip right to the interesting bits:

In more than one scenario, several configurations are able to handle at least 60k qps for 10 minutes with room to spare. Finding their upper limit is the immediate next goal.

Let us examine a few characteristic results, to better orient ourselves:

One of the biggest pitfalls when referring to examples is they might not be optimized for our use case.

Yes, it was written clearly in reitit's and muuntaja's documentation. Do you always refer back to the documentation when you already think you know?

First order of business is pushing each configuration to its breaking point, just so see how far we can go. That's before doing any optimization or tuning.

Next will be a post which explains the profiling process and automation the performance space search, saving the need to manually set the run rates.

Once the profiling process is fully automated, I'll be able to throw in additional JDKs into the mix.

On a different track, the results of these experiments have already birthed several PRs to ring, reitit and http-kit. Once they are merged I can rerun all the experiments with them and get new and improved results. My poor machine.

Finally, there is certainly room for informed design optimizations, such as thread pools assignment. It's possible some servers have been using the same pool for the event loop and processing requests which degraded their results.