Jepsen

Bugbash 2026 - Paranormal Forms

Jepsen — Thu, 26 Mar 2026 00:00:00 GMT

We have several tricks for efficiently checking the consistency of transactions over primary keys, but what are we to do about predicates? Database histories with predicates are particularly tricky because their correctness hinges on the entire universe of objects, not just those objects which matched the predicate. When a query for all even objects returns the empty set, is that because it selected the unborn version of x, or one in which x = 1? There are many possibilities, each resulting in a different dependency graph. The proliferation of these graphs may be intractable, unless we find a way to prove exactly which invisible version was selected.

If you’d like to talk about this problem, come to Bugbash 2026 on April 22-24th, in Washington, DC. Jepsen’s seminar, “Paranormal forms”, will introduce the problem of predicate checking, suggest a few possibilities, then open the floor to discussion.

MariaDB Galera Cluster 12.1.2

Jepsen — Mon, 16 Mar 2026 00:00:00 GMT

Jepsen tested MariaDB Galera Cluster, versions 12.1.2 through 12.2.2, and found two scenarios which led to the loss of committed transactions. First, under the recommended configuration settings it does not flush data to disk before acknowledgement; committed transactions can be lost when nodes crash in quick succession. Second, it occasionally loses committed transactions with process crashes and network partitions. Even without faults, MariaDB Galera Cluster allows P4 (Lost Update), and therefore fails to satisfy its claimed isolation level “between Serializable and Repeatable Read”. It also regularly exhibits Stale Read.

NATS 2.12.1

Jepsen — Mon, 08 Dec 2025 00:00:00 GMT

NATS is a popular distributed streaming system. Jepsen tested NATS 2.12.1, focusing on its durable JetStream subsystem, and found that it could lose data or get stuck in persistent split-brain in response to file corruption or simulated node failures. This data loss was caused in part by a default fsync policy which flushed data to disk once every two minutes, rather than before acknowledgement. Even a single kernel crash or power failure, combined with process pauses or network partitions, could cause NATS replicas to lose acknowledged messages. NATS has documented its default lazy fsync setting, and is considering the other issues we found.

Jepsen 0.3.10

Jepsen — Tue, 02 Dec 2025 00:00:00 GMT

A new Jepsen release, 0.3.10, is now available on GitHub and Clojars. This release is aimed at controllable entropy and support for running Jepsen inside Antithesis: a deterministic simulation testing environment. A new supporting library, jepsen.generator, provides the current generator system along with jepsen.random: a new namespace for pluggable random value generation. Jepsen uses these RNGs throughout, which makes it possible to run a test with a deterministic seed, or to source entropy from an external system, like Antithesis. The jepsen.antithesis library provides additional support for assertions, randomness, and lifecycle operations, plus wrappers for clients and checkers.

Also, this release introduces a new kind of visualization: op color plots, which show operations over time with different user-defined colors. This is particularly helpful for getting a feeling for “when did we lose data?” or “did only read-only queries succeed during a partition?”

A Distributed Systems Reliability Glossary

Jepsen — Mon, 20 Oct 2025 00:00:00 GMT

Jepsen and Antithesis wrote A Distributed Systems Reliability Glossary: a free reference for engineers who build, test, and operate distributed systems. It covers basic concurrency theory, consistency models, various faults, approaches to testing, and offers some links to further reading.

Jepsen 18: Serializable Mom

Jepsen — Mon, 18 Aug 2025 00:00:00 GMT

The latest Jepsen talk, “Jepsen 18: Serializable Mom”, is now available on Youtube. This talk was presented on June 20, 2025, at Systems Distributed in Amsterdam. It covers Bufstream 0.1.0, Amazon RDS for PostgreSQL 17.4, and TigerBeetle 0.16.1.

Capela dda5892

Jepsen — Thu, 07 Aug 2025 00:00:00 GMT

Jepsen and Capela, Inc worked together to test early builds of Capela, an unreleased distributed programming environment. Our analysis found twenty-two issues, including four problems in Capela’s programming language semantics, fourteen crashes or non-fatal panics, severe performance degradation after roughly a minute of operation, and three safety issues: partitions ignoring their initial values, sporadically vanishing, and losing committed writes. Capela has fixed two of the language issues—the others remain under investigation.

The GeekNarrator Podcast

Jepsen — Wed, 06 Aug 2025 00:00:00 GMT

Kaivalya Apte interviewed Kyle Kingsbury for The GeekNarrator Podcast. We talk about common bugs in distributed systems, type I vs type II errors, ensuring correctness in Jepsen itself, LLMs, experimental techniques, formal verification, and more.

Jepsen 17: ACID Jazz

Jepsen — Tue, 05 Aug 2025 00:00:00 GMT

Jepsen’s 17th conference talk, “ACID Jazz”, is now available on YouTube. This talk was presented at Antithesis’ BugBash conference, in Washington, DC, and covers research on MySQL 8.0.34, Datomic Pro 1.0.7075, and Bufstream 0.1.0.

Distributed Systems Glossary

Jepsen — Tue, 15 Jul 2025 00:00:00 GMT

Antithesis and Jepsen are proud to present a glossary for distributed systems reliability. We hope this will be helpful for engineers, testers, and students!

TigerBeetle 0.16.11

Jepsen — Fri, 06 Jun 2025 00:00:00 GMT

TigerBeetle is a distributed OLTP database for financial transactions. We worked with TigerBeetle to test versions 0.16.11 through 0.16.30, and found seven crashes, elevated latencies during single-node failures, and requests which were retried forever. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a debugging API. As of version 0.16.45, TigerBeetle had addressed every issue, except for indefinite retries.

Amazon RDS for PostgreSQL 17.4

Jepsen — Tue, 29 Apr 2025 00:00:00 GMT

With a new experimental library for running Jepsen tests on Amazon RDS clusters, we report on a small issue in Amazon RDS for PostgreSQL. At the “Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4. Amazon RDS for PostgreSQL may instead support Parallel Snapshot Isolation, a slightly weaker consistency model.

Upcoming Events

Jepsen — Tue, 04 Mar 2025 00:00:00 GMT

By popular demand, we’re offering another open session of the distributed systems fundamentals class: four half-days discussing the basics of distributed systems theory and practice. For the first time we’re also opening up the accompanying workshop, for up to five participants. Please join!

Jepsen also has two conference talks coming up: one at BugBash 2025 (April 3-4), in Washington, DC, and a second at Systems Distributed 2025 (June 19-20), in Amsterdam.

Antithesis & Bufstream Webinar

Jepsen — Wed, 27 Nov 2024 00:00:00 GMT

Antithesis, Buf, and Jepsen are running a joint webinar on December 5th, 2024. We’ll discuss a Kafka protocol safety issue, talk about the challenges of distributed systems testing, and show how Jepsen and Antithesis helped identify critical safety errors in Bufstream. Come watch Antithesis pause, rewind, and explore a running Bufstream cluster in an interactive debugging shell!

Bufstream 0.1.0

Jepsen — Fri, 25 Oct 2024 00:00:00 GMT

Jepsen worked with Buf to analyze the safety of Bufstream, a Kafka-compatible streaming system. We found three safety and two liveness issues in Bufstream 0.1.0, including the loss of acknowledged writes in healthy clusters. These problems were resolved by version 0.1.3. We also discovered serious issues in Kafka’s transaction protocol, including write loss, aborted read, and torn transactions. These problems affect Kafka, Bufstream, and (presumably) other Kafka-compatible systems; they remain outstanding.

MariaDB Snapshot Isolation

Jepsen — Fri, 25 Oct 2024 00:00:00 GMT

In late 2023 we reported that MySQL and MariaDB’s REPEATABLE READ did not, in fact, provide repeatable reads. The MariaDB team has been hard at work this past year. They’ve added a new flag, --innodb-snapshot-isolation=true, which causes REPEATABLE READ to prevent Lost Update, Non-repeatable Read, and violations of Monotonic Atomic View.

Jepsen has not yet tested this, but it looks like MariaDB might, with the new flag enabled, offer Snapshot Isolation at REPEATABLE READ.

Distributed Systems Fundamentals - Open Session

Jepsen — Fri, 25 Oct 2024 00:00:00 GMT

Jepsen’s distributed systems training introduces engineers and operators to the fundamentals of nodes and networks, consistency and availability, techniques for replicating state, a slew of design patterns, and production concerns. By popular request, we’re offering a special session of this class that anyone can register for. Join us on Zoom, December 16th through 19th, 2024. Tickets are on sale now.

Jepsen 0.3.6

Jepsen — Fri, 18 Oct 2024 00:00:00 GMT

Jepsen 0.3.6 is now available on GitHub and Clojars. This is a sizeable release. It includes a significant correctness bugfix for a rare bug that could make operations in the history print with the wrong data. It also adds a new namespace for composing databases, nemeses, and generators when working with systems where each node has a different role. Kafka-style tests gain new powers and are significantly faster. And we have the usual slew of small bugfixes, dependency bumps, and quality-of-life improvements. Happy testing!

The full release notes are available on GitHub.

Jepsen 16 - Systems Distributed 2024

Jepsen — Thu, 29 Aug 2024 00:00:00 GMT

Kyle Kingsbury spoke at Systems Distributed 2024 on RavenDB, MariaDB/MySQL, and Datomic.

jetcd 0.8.2

Jepsen — Thu, 08 Aug 2024 00:00:00 GMT

Jepsen traced lost update, circular information flow, and aborted reads in etcd tests to an improper retry mechanism in jetcd 0.8.2, which allowed transactions to be submitted multiple times, and for committed transactions to appear as if they had actually failed.

Goto Chicago 2024

Jepsen — Tue, 23 Jul 2024 00:00:00 GMT

Kyle Kingsbury will speak on performance techniques in Jepsen at GOTO Chicago, October 21 & 22, 2024. The talk will touch on a mix of high-level and low-level performance optimizations to make checking large histories tractable, including parallelism, pure functions, immutable data structures, and deforestation; bitsets, avoiding sharing between threads, packing structures into mutable arrays, dynamic compilation of primitive boxes, and macro iteration magic.

Early bird tickets are on sale now.

Ethics Policy Update

Jepsen — Wed, 17 Jul 2024 00:00:00 GMT

We’ve made some small changes to the Jepsen ethics policy.

The policy used to promise that Jepsen could veto publication if Jepsen and a client could not agree on the content of an analysis. However, this veto has never been used. In fact, Jepsen’s contracts have given Jepsen final approval over the content of analyses since 2016. We replace the promise of a veto with a stronger promise of editorial control.

In light of Jepsen’s multiple authors, we also shift to an organizational third person voice. Finally, we’ve streamlined some language.

Datomic Pro 1.0.7075

Jepsen — Wed, 15 May 2024 00:00:00 GMT

In collaboration with Nubank, we analyzed Datomic Pro 1.0.7075 and found that its inter-transaction safety properties appeared stronger than claimed. Datomic Pro appeared to offer Strong Session Serializable isolation, and Strong Serializable for histories restricted to update transactions. However, Datomic defines unusual intra-transaction semantics in which operations are applied logically concurrent with one another, rather than sequentially. While consistent with Datomic’s documentation, this could cause invariants preserved by individual transaction functions to be broken when those same functions are applied within a single transaction.

RavenDB 6.0.2

Jepsen — Wed, 31 Jan 2024 00:00:00 GMT

In a brief survey of RavenDB 6.0.2, we found “ACID” transactions allowed both lost updates and fractured read, even in healthy single-node clusters. Depending on how you interpret RavenDB’s documentation and response to this work, RavenDB may not have interactive transactions at all.

MySQL 8.0.34

Jepsen — Tue, 19 Dec 2023 00:00:00 GMT

We revisited Kleppmann’s work on MySQL isolation levels and found surprising behavior in 8.0.34. MySQL’s REPEATABLE READ not only exhibits G2-item, G-single, and lost update, but also violates internal consistency and Monotonic Atomic View. It satisfies neither Adya’s Repeatable Read nor the ambiguous ANSI SQL definition. We also discovered AWS RDS MySQL clusters routinely violate Serializability at the SERIALIZABLE isolation level.

Redpanda 21.10.1

Jepsen — Fri, 29 Apr 2022 00:00:00 GMT

Jepsen and Redpanda worked together to analyze the Redpanda distributed queue, versions 21.10.1 through 21.11.2. We found three liveness and seven safety issues, including crashes, aborted reads, inconsistent offsets, circular information flow, and lost or stale messages. We also discuss surprising aspects of the Kafka/Redpanda transaction model. Redpanda has resolved seven of the issues we found, although lost/stale messages and one crash remain under investigation. A few other issues require only documentation to clarify intended behavior.

Radix DLT 1.0-beta.35.1

Jepsen — Sat, 05 Feb 2022 00:00:00 GMT

We analyzed the Radix DLT distributed ledger system at version 1.0-beta.35.1, 1.0.0, 1.0.1, and 1.0.2. We found 11 safety errors including stale reads, aborted and intermediate reads, and the partial or total loss of committed transactions from transaction log. We also identified issues with transactions which hung indefinitely, and degraded performance under single-node faults. Following a system redesign RDX Works reports all safety issues, as well as the indefinite transaction issue, were fixed in version 1.1.0.

Scylla 4.2-rc3

Jepsen — Wed, 23 Dec 2020 00:00:00 GMT

Together with the ScyllaDB team, we found seven problems in Scylla, including lightweight transaction (LWT) split-brain in healthy clusters due to a.) incomplete row hashcodes and b.) multiple problems with membership changes. We also identified incomplete or inaccurate documentation, including claims that non-LWT operations were isolated and atomic, and undocumented rules about what kinds of membership operations were legal. Scylla has corrected almost all of these errors via patches and documentation; remaining cases of split-brain appear limited to concurrent membership changes

Redis-Raft 1b3fbf6

Jepsen — Tue, 23 Jun 2020 00:00:00 GMT

We helped Redis Labs verify early development builds of Redis-Raft, and found twenty one issues, including crashes, split-brain, infinite loops, aborted reads, data corruption, and total data loss on any failover. Redis Labs has addressed all but one of these issues in recent versions, and continues to work towards general availability in 2021.

PostgreSQL 12.3

Jepsen — Fri, 12 Jun 2020 00:00:00 GMT

Jepsen identified a serializability violation in PostgreSQL 12.3, where concurrent inserts and updates could result in transactions which fail to observe each other’s effects. This bug appears to have been present since the implementation of serializable snapshot isolation in version 9.1. A patch should be available in the next minor release, currently scheduled for August 13th.

MongoDB 4.2.6

Jepsen — Fri, 15 May 2020 00:00:00 GMT

A brief investigation into MongoDB 4.2.6’s transaction system found violations of snapshot isolation, rather than claimed “full ACID” guarantees. Weak defaults allowed transactions to lose writes and allow stale reads unless carefully controlled.

Dgraph 1.1.1

Jepsen — Thu, 30 Apr 2020 00:00:00 GMT

Dgraph Labs, makers of the Dgraph graph database, worked with Jepsen to follow up on our 2018 analysis of Dgraph 1.0.2. We are pleased to report that Dgraph Labs resolved all issues from this previous work, and together, we identified five new safety issues in version 1.1.1, all involving tablet migration. Three of these issues appear addressed, and the remainder are under investigation.