We have several tricks for efficiently checking the consistency of transactions over primary keys, but what are we to do about predicates? Database histories with predicates are particularly tricky because their correctness hinges on the entire universe of objects, not just those objects which matched the predicate. When a query for all even objects returns the empty set, is that because it selected the unborn version of x, or one in which x = 1? There are many possibilities, each resulting in a different dependency graph. The proliferation of these graphs may be intractable, unless we find a way to prove exactly which invisible version was selected.

If you’d like to talk about this problem, come to Bugbash 2026 on April 22-24th, in Washington, DC. Jepsen’s seminar, “Paranormal forms”, will introduce the problem of predicate checking, suggest a few possibilities, then open the floor to discussion.

Jepsen tested MariaDB Galera Cluster, versions 12.1.2 through 12.2.2, and found two scenarios which led to the loss of committed transactions. First, under the recommended configuration settings it does not flush data to disk before acknowledgement; committed transactions can be lost when nodes crash in quick succession. Second, it occasionally loses committed transactions with process crashes and network partitions. Even without faults, MariaDB Galera Cluster allows P4 (Lost Update), and therefore fails to satisfy its claimed isolation level “between Serializable and Repeatable Read”. It also regularly exhibits Stale Read.

NATS is a popular distributed streaming system. Jepsen tested NATS 2.12.1, focusing on its durable JetStream subsystem, and found that it could lose data or get stuck in persistent split-brain in response to file corruption or simulated node failures. This data loss was caused in part by a default fsync policy which flushed data to disk once every two minutes, rather than before acknowledgement. Even a single kernel crash or power failure, combined with process pauses or network partitions, could cause NATS replicas to lose acknowledged messages. NATS has documented its default lazy fsync setting, and is considering the other issues we found.

A new Jepsen release, 0.3.10, is now available on GitHub and Clojars. This release is aimed at controllable entropy and support for running Jepsen inside Antithesis: a deterministic simulation testing environment. A new supporting library, jepsen.generator, provides the current generator system along with jepsen.random: a new namespace for pluggable random value generation. Jepsen uses these RNGs throughout, which makes it possible to run a test with a deterministic seed, or to source entropy from an external system, like Antithesis. The jepsen.antithesis library provides additional support for assertions, randomness, and lifecycle operations, plus wrappers for clients and checkers.

Also, this release introduces a new kind of visualization: op color plots, which show operations over time with different user-defined colors. This is particularly helpful for getting a feeling for “when did we lose data?” or “did only read-only queries succeed during a partition?”

Jepsen and Antithesis wrote A Distributed Systems Reliability Glossary: a free reference for engineers who build, test, and operate distributed systems. It covers basic concurrency theory, consistency models, various faults, approaches to testing, and offers some links to further reading.