Debugging QUIC with H2O and QLog

One of the key qualities of a computer program is its debuggability. This is especially true for protocol implementations where interoperability with different implementations comes into play. In addition to testing for correctness, good debuggability also helps identify opportunities for performance tuning. This is crucial for QUIC, given how the protocol currently needs to be implemented in user-space. In this blog post, I will cover how we added support for QLog, an incremental QUIC endpoint logging format, to H2O, an open-source HTTP server that we deploy throughout our edge cloud platform.

QUIC, QLog, and debugging tools

QUIC is a secure low-latency transport layer protocol that is used for HTTP/3. QUIC is an exciting new protocol that addresses fundamental problems that TCP faces in today's internet environment. If this also excites you, take a look at the QUIC-related series of blog posts by our distinguished engineer, Jana Iyengar.

QLog is a QUIC endpoint logging format. The QLog project also provides QVis, a JavaScript based visualization toolsuite for QLog data. While QLog and QVis are often discussed together, it's important to note that the QLog logging format is useful even without the visualizer, since it now offers a standard logging format that we can build tooling around. This is why we support the QLog project: a de-facto standard language for QUIC tracing data would benefit not only developers, but the wider internet community at large. The QLog specification is currently evolving alongside QUIC. 

Reusing H2O's event tracing infrastructure

The H2O server provides a comprehensive low-level event tracing feature, which is powered by eBPF or DTrace, depending on the platform. To be more specific, event tracing is accomplished by instrumenting the H2O server and its underlying libraries (e.g. quicly, H2O's QUIC implementation) with USDT (User Statically Defined Tracing) probes. These probes are production friendly, since the overhead of an inactive probe is just a single NOP instruction. Given this tracing infrastructure, we wrote a post-processing QLog adapter that enabled us to support the QLog format while retaining H2O's powerful low-level tracing capabilities. Using an adapter also allows us to consider different logging formats and tools in the future without changing the tracing infrastructure in H2O.

The difficult work was handling structural differences between H2O’s low-level tracing and QLog's higher-level semantics. For example, packet and stream events are distinctly handled in H2O, but they are tightly coupled in the QLog format. We eventually reached a flexible design that allowed us to accomplish this mapping.

The future of QUIC

The days ahead are looking brighter for QUIC. The IETF standardization process is nearing the finish line, the Google Chrome browser now officially supports IETF QUIC, and here at Fastly, we have deployed QUIC and HTTP/3 across our entire server fleet. As a result, we are seeing increasing QUIC traffic as we bring more customers on board. Our new ability to log and examine QUIC endpoint states via QLog will help our protocol engineering team to build a more secure and snappier internet.