Field Notes — Issue Archive

644 dispatches from the systems that bend, break, and behave oddly.

A reverse-chronological log of postmortems, deep dives, and oddities — each one a small experiment in understanding what happens when well-built things encounter the real world. Filter by category, severity, or tag.

Browse the archive About the lab

644Dispatches

38System Categories

12Years Documented

0Clean Systems Found

Reverse-Chronological Index

The complete field notes archive

Every entry is a real observation from a real system. Newest first. Filter the noise out, dig into the signal.

Browse by topic

Featured

Mar 18, 2026 14 min read ● High Severity

When the cache outlived the request: a 47-minute degradation in a payments-routing tier

The cache was technically correct. The request was technically served. And yet for 47 minutes, a regional routing tier quietly served 18% of payment intents from a stale snapshot that no longer matched the rate-limiter's worldview. A postmortem on correctness that wasn't wrong, exactly — just out of phase with reality.

#postmortem #cache-coherence #payments #distributed-state

Read the full dispatch

Distributed Systems

Mar 14, 2026 ● Medium

The case of the 412ms clock skew that nobody noticed for six months

An NTP drift in a single availability zone produced a quiet but durable ordering violation in a workload-distribution layer. The system kept working. The audits kept passing. The data was just subtly, persistently wrong.

9 min · #time · #ntp Read