Long-form investigations into systems that misbehave.
A curated archive of reproducible case studies — each one traces a real failure, a quiet oddity, or an unintended behavior back to the mechanism that produced it. Bring your own hypothesis; we'll show the receipts.
A working catalogue of systems that bent, broke, and behaved oddly.
Six reproducible deep-dives drawn from the archive. Each one is a complete lab notebook — symptoms, instrumentation, the moment of clarification, and the smallest patch that fixed the rest.
When clocks lie: tracing a 4.7 ms drift across three datacenters
How a single NTP misconfiguration caused a Raft-based metadata store to elect a phantom leader every 19 hours — and the one line of Prometheus alerting that caught it before users did.
Read the guideThe 2.6× throughput mystery: tcp_rmem, autotuning, and a kernel commit from 2017
An egress proxy looked bottlenecked at 8 Gbps on a 25 Gbps NIC. eBPF probing revealed the receive window was collapsing under sustained 64-packet bursts. The fix lived in /proc — not in our app.
Read the guideAn optimizer that ordered the wrong rows: a Postgres planner walkthrough
Two semantically equivalent queries returned in 12 ms and 9,400 ms. We disassembled both plans, ran the planner in debug mode, and watched it pick a nested loop over a hash join it had every reason to choose.
Read the guideSpectre, four years later: measuring residual leaks on patched hardware
We built a deterministic microbenchmark to detect cross-process cache residue on three generations of server CPUs. Two of them still leaked — just much more politely. Includes the harness, raw data, and a containment checklist.
Read the guide