How CockroachDB survives anything

One logical database, many machines

A cockroach is famously hard to kill. The database borrows the name because its central promise is survivability: lose a disk, a node, a rack, an availability zone — even an entire region — and the database stays available and consistent. You keep talking to it in ordinary SQL over the PostgreSQL wire protocol; it quietly handles replication, failover, and rebalancing.

It pulls this off by never trusting a single machine with anything. Every piece of data lives on at least three nodes, and a write is only acknowledged once a majority of those nodes agree on it. That single idea — majority agreement — is what lets a node die mid-write without losing data or serving a stale read.

Everything is one big sorted map

CockroachDB stores your entire cluster — every table, index, and system record — as a single, monolithic, sorted map of key→value pairs. There are no per-table files. The SQL layer's job is to encode rows into keys in that map, choosing the encoding so that rows you tend to read together end up next to each other in key order.

A row's primary-key value becomes part of its key. Take a table users(id PRIMARY KEY, email, name). The row for id = 42 is stored roughly like this:

Because keys are sorted, a range scan like WHERE id BETWEEN 42 AND 99 is a contiguous read. A secondary index is just more keys, encoded so the index column sorts first. This is why the physical layout of a CockroachDB table is really a question about key order.

Ranges: the unit of everything

The sorted keyspace is chopped into contiguous chunks called ranges. A range holds a band of keys — say [/users/1, /users/9000) — and is the atom of replication, distribution, and load-balancing. By default a range aims to stay under 512 MiB; when it grows past that it splits in two, and when adjacent ranges shrink they can merge. Ranges also split under load, so a hot key band can spread across more nodes.

A tiny, always-present meta range acts as an index of ranges: given a key, it tells a node which range holds it and where that range's replicas live. Nodes cache this, so routing a query to the right machine is usually a local lookup.

Raft: how three replicas agree

Every range is replicated — three copies by default (five for critical system ranges). Those replicas form a Raft group. Raft is a consensus protocol whose whole purpose is to keep a replicated log identical across machines, even as they crash and recover. At any moment one replica is the leader; it accepts new log entries and replicates them to the followers.

A write becomes durable when a majority (a quorum) has appended it to their log. With three replicas the quorum is two, so one node can fail with zero data loss and no downtime for that range. The surviving majority keeps committing; when the dead node returns, Raft streams it the entries it missed.

Leaseholders make reads cheap

If every read had to run a full round of Raft, reads would be as expensive as writes. They don't. For each range, one replica holds a time-bounded lease. That leaseholder has the authoritative, up-to-date copy, so it can answer reads locally — no quorum, no network round trip to the followers — as long as it reads at or below its own clock and respects outstanding writes.

Writes still go through Raft (a majority must persist them), but the leaseholder coordinates them and applies them in order. Leases are kept alive cheaply using node liveness: a single heartbeat range records that a node is alive, and one heartbeat covers all of that node's leases at once. If a leaseholder dies, its leases expire and another replica picks them up — the same failover you saw in the Raft simulator, one layer up.

Hybrid logical clocks: making time monotonic

Here's the hard part of any distributed database: ordering events across machines whose clocks disagree. Physical wall clocks drift. If node A's clock is 30 ms ahead of node B's, then an event that "happened after" another could carry an earlier timestamp — and your transactions would see time run backwards.

CockroachDB's answer is the Hybrid Logical Clock (HLC). Each node's clock is a pair:

// an HLC timestamp is two numbers
timestamp = ( physical: wall-clock millis , logical: a counter )

The physical part tracks real time so timestamps stay close to the wall clock (useful for TTLs and AS OF SYSTEM TIME). The logical counter breaks ties and preserves causality: it only ever goes up. The rules are simple and they guarantee timestamps never move backwards:

• Local event: take max(physical, now()); if physical didn't advance, bump the logical counter.
• Send a message: attach your current HLC.
• Receive a message: set your clock to max(mine, theirs), then tick. You can never look older than a message you've already seen.

Why not Google Spanner's TrueTime?

Spanner uses TrueTime — GPS and atomic clocks in every datacentre — so it can bound clock error to a few milliseconds and simply wait out the uncertainty on commit. CockroachDB is designed to run on commodity hardware with ordinary NTP, so it can't assume tight clock bounds. Instead of special hardware it uses the HLC plus an explicit uncertainty interval, described next.

Clock uncertainty & the read that restarts

Because clocks aren't perfectly synced, CockroachDB assumes any two nodes are within a maximum clock offset of each other — 500 ms by default (--max-offset). This has two consequences, one for safety and one for liveness.

That safe thing is an uncertainty restart: the transaction bumps its read timestamp up past the surprising value and retries the read, now certain it will see it. Crucially, the uncertainty window shrinks with each restart and never grows, so this can't loop forever. The payoff is a strong guarantee: no stale reads for causally-related transactions, without atomic clocks.

MVCC: writes never overwrite, timestamps only rise

CockroachDB never overwrites a value in place. Every write stores a new version of the key, stamped with the transaction's commit timestamp. The key in Pebble is really (key, timestamp), and versions are sorted newest-first. This is Multi-Version Concurrency Control (MVCC), and it's why the earlier claim — "time only moves forward" — is literally true in the storage engine.

A read at timestamp T walks down to the newest version with version_ts ≤ T. That gives three superpowers for free:

• Non-blocking reads: a reader at T ignores newer versions instead of waiting on writers.
• Snapshot consistency: a transaction sees one coherent instant of the whole database.
• Time travel: SELECT ... AS OF SYSTEM TIME '-10s' just reads at an older timestamp.

Old versions don't pile up forever. A background MVCC garbage collector reclaims versions older than the zone's gc.ttlseconds — 25 hours by default — as long as no protected timestamp (e.g. a running backup or changefeed) still needs them. That 25-hour window is also the horizon for how far back AS OF SYSTEM TIME can travel.

Distributed transactions, serializable by default

Now the pieces combine. CockroachDB runs transactions at SERIALIZABLE isolation by default — the strongest level, with no anomalies — across keys that may live on completely different machines. (A weaker READ COMMITTED level is also available when you want fewer retries and can tolerate its anomalies.) It does this without a central lock manager.

The life of a write transaction

1. Pick a timestamp. The gateway node stamps the transaction with its HLC — this becomes the provisional commit timestamp.
2. Lay down write intents. Each write is stored as a provisional MVCC version — a write intent — that points at a single transaction record. An intent is a value and a lock in one.
3. Resolve conflicts. A reader or writer that meets an intent looks up the transaction record: if that txn committed, the intent is promoted to a real value; if it's still pending, the newcomer may wait or force a restart.
4. Commit. Flip the transaction record to COMMITTED. Parallel Commits lets the intents and the record be written in one round of consensus instead of two, roughly halving commit latency.
5. Clean up. Asynchronously rewrite intents into normal MVCC versions. Readers that arrive first will do it for you.

Serializability is enforced with a timestamp cache (remembering the highest timestamp at which each key was read, so a later write can't invalidate that read) and the uncertainty logic from earlier. When a write would break serial order, the transaction's commit timestamp is pushed forward; if that push can't be proven safe, it restarts at the higher timestamp. Every timestamp you saw rising in the HLC and MVCC demos is what makes this ordering possible.

Gossip, rebalancing & multi-region

Zone configurations are the knobs for all of this: num_replicas, constraints (place replicas in region=eu-west), lease_preferences (keep the leaseholder near your users for fast reads), and gc.ttlseconds. You can set them per cluster, database, table, index, or even row range.

Talk to a real cluster

Everything above runs entirely in your browser. This last panel talks to an actual CockroachDB cluster through the Worker's /api endpoints — reading its live hybrid logical clock, listing the real nodes in its gossip network, and showing how its keyspace is split into ranges. It's read-only, and it's dormant until a cluster is attached.