HomeTechnical ArchitecturePhilosophyContactRequest access
All articles

GroupThink

Mechanism: Clocks You Cannot Trust

20 min read

Two of your devices edit the same note a minute apart, and one of them has a clock that runs four minutes fast. Under the obvious merge rule — newest timestamp wins — the wrong edit survives, nothing raises an error, and you find out weeks later, if at all. The repair is to stop asking when an edit happened and start asking what it had already seen: one small integer per device, and a comparison that never reads a clock.

Prerequisites: Mechanism: Reconciling by Digest for how a divergence gets found in the first place, and Staying in Sync for the loop this merge rule sits inside.

1. An edit that was never there

The digest round tells you that a namespace diverged and hands you the rows that differ. It does not tell you which version is right — and here that decision is entirely local, because no server holds an authoritative copy to defer to.

The obvious rule is the one almost every sync layer reaches for first. Every mutable row already carries an updatedAt; on conflict, keep the larger. Last-write-wins. It was the rule here too, in a function called mergeNewestWins, until the commit that generalized causal merge deleted it.

Here is what it does. Your iPad's clock is four minutes fast. At 10:00 you jot a note on the iPad; it is stamped 10:04. At 10:02 you open the same note on your iPhone and rewrite the second half; it is stamped 10:02. The iPhone edit is genuinely later — it happened after, and it was made by someone who had just read the iPad's version — and it carries the smaller number. When the two devices reconcile, the iPad's text wins and the iPhone's rewrite is deleted.

WHAT HAPPENED WHAT THE MERGE SEES RULE: NEWEST STAMP WINS REAL TIME IPAD — WRITES V1 REAL 10:00 CLOCK RUNS +4 MIN IPHONE — REWRITES V2 REAL 10:02 · SAW V1 GENUINELY LATER ORDER INVERTED BY SKEW V2 — STAMPED 10:02 SMALLER NUMBER REWRITE DELETED · NO ERROR V1 — STAMPED 10:04 LARGEST NUMBER STALE TEXT KEPT WHICHEVER CLOCK RUNS FAST WINS — AND THE LOSS HAS NO SYMPTOM

Notice what does not happen. No conflict object is created, no error is logged, no second copy is kept for you to resolve. From inside the merge function, "the incoming row is older, keep mine" is byte-for-byte the same code path as "nothing to do here" — the overwhelmingly common case, and the one every test exercises. The loss has no symptom. You will not notice until you go looking for a paragraph you are sure you wrote.

Two separate things are wrong here. Even with perfect clocks, last-write-wins throws away one of two edits; that is a known and sometimes acceptable cost. With skew, it throws away the edit you expected to keep, and the effective rule becomes whichever device's clock runs fast wins — which no user can predict and no test will reproduce.

And clocks have the worst possible property for a correctness argument: they are usually fine. A phone on a network is NTP-synchronised to within milliseconds, so the bug is invisible in development, in review, and for most users most of the time. It surfaces on the device whose battery died and whose real-time clock reset, or the one a user set forward by hand. A guarantee that holds "usually" is a latent bug with a long fuse.

2. Counting instead of timing

Back up and ask what the merge actually needed to know. Not what time it was: whether the device that made the second edit had already seen the first. If it had, the second edit supersedes the first and there is no conflict at all — only a sequence. If it had not, the two are a genuine fork, and no timestamp can make either legitimately newer, because neither device knew the other existed. "Had seen" is a claim about information flow, and information flow can be counted.

A version vector is a map from a device identifier to a monotonically increasing integer. In VersionVector.swift it is exactly that and nothing else — counters: [String: Int] — and every row in a namespace that needs causal merge carries one. A missing key means zero. Two operations define the whole semantics:

Bump your own dimension, never anyone else's. bump(_ device:) does counters[device, default: 0] += 1, and the only device name it is ever called with is this device's own. That restriction is the entire integrity story. Your counter counts changes you made; the other entries record what you have received. If a device could write another device's number, it could claim to have seen edits that were never delivered to it, and every later comparison would believe the claim.

Merge by elementwise maximum. merged(with:) takes, for each device, the larger of the two counts — reading a sibling's row means you now know everything it knew plus everything you knew, and the elementwise max is exactly that union. It is commutative, associative, and idempotent, which is what lets the reconcile run from both ends independently without the two runs fighting.

Comparison falls out of those two rules. relation(to:) walks the union of both key sets and tracks two flags — is there a dimension where I am ahead, and is there one where you are? — which gives four outcomes: .dominates (ahead somewhere, behind nowhere), .dominated (the mirror), .equal, and .concurrent (ahead somewhere and behind somewhere). If my vector dominates yours, every change you have counted I have counted too, plus at least one you have not — and the only way I could have obtained your counts is by merging your row or by making those changes myself. My version therefore descends from yours: a proof of ordering built from integers and a > operator, with no clock anywhere in it.

One detail decides whether the dimensions mean anything: what a device calls itself. SettingsSyncClock.configure(deviceID:) pins it to the device's envelope public key in hex, so the axes are the real fleet, and until that happens the clock refuses to bump at all — because if it bumped under the throwaway UUID it mints as a fallback, a later configure would orphan every vector recorded under the old name, splitting one physical device into two axes that look permanently concurrent.

3. The same edit, worked through

Take the scenario from §1 again, this time with vectors. There is a test for it — notebookCausalEditBeatsWallClock — and the trace below is what it asserts.

Device A creates a note titled v1 and stamps updatedAt at wall-clock time 100. RowClock.observe sees a row with no bookkeeping entry, registers one, and bumps A's dimension. B then pulls the notebook namespace from A; B has no local row, so there is nothing to resolve — it inserts the note and records the merged vector, which is A's:

A: "v1"  updatedAt=100  {A:1}
B: "v1"  updatedAt=100  {A:1}

Now B edits the note to v2. B's clock is behind, so the new updatedAt is 50 — earlier than the value already on the row. On the next save, RowClock.observe notices that the live row's updatedAt no longer matches the syncedAt it last recorded, and bumps B's own dimension:

A: "v1"  updatedAt=100  {A:1}
B: "v2"  updatedAt=50   {A:1, B:1}

A pulls from B and compares {A:1} against {A:1, B:1}. Dimension A: 1 versus 1, neither ahead. Dimension B: 0 versus 1, the incoming vector is ahead. So selfGreater is false, otherGreater is true — .dominated. A adopts v2.

That is the right answer, reached without either device reading a clock. B's version wins because it descends from A's. The updatedAt values, 100 against 50, played no part; had they been consulted, A would have kept v1 and the rewrite would have vanished.

DEVICE A DEVICE B NOTE — V1 NOTE — V2 UPDATEDAT = 100 { A:1 } { A:1 · B:1 } NOTE — V2 UPDATEDAT = 50 { A:1 · B:1 } DESCENDS FROM A'S ROW A PULLS FROM B RELATION(TO:) DIM A · 1 VS 1 · NEITHER AHEAD DIM B · 0 VS 1 · OTHER AHEAD .DOMINATED — A ADOPTS V2 THE 100-VS-50 STAMPS PLAYED NO PART ORDERING FROM INTEGERS AND > — NO CLOCK ANYWHERE IN IT

Change one thing and the answer changes completely. Suppose B never pulled A's note and instead materialised its own row under the same identifier. Then B's vector is {B:1}, A's is {A:1}, and the comparison finds each ahead on its own dimension: .concurrent.

4. Detected is not resolved

That last case is where a version vector stops helping, and it is worth being blunt about it, because the machinery has a way of feeling like it solves conflicts. It does not. It classifies them. Three of its four answers are decisions; the fourth is a question handed back to the caller. (.equal belongs with .concurrent here, and the code groups them: two rows with identical vectors and different content have no causal relationship either.)

VersionVector.resolveMerge holds the one policy the whole system shares. Where causality decides, it obeys causality. Where it does not, two rules apply:

  • Add-wins on presence. A live value beats a tombstone; two tombstones stay deleted. A concurrent delete and edit resolve toward the data existing. (Deletion has a whole article of its own — Deleting Things That Stay Deleted.)
  • A caller-supplied tie-break between two live values. For rows it is lexGreater over the canonically encoded entry bytes; for settings it is the lexicographically larger string value.

Both tie-breaks are deterministic and clock-free, which is all convergence requires: every device, shown the same two candidates, picks the same winner. But be clear about what is purchased. The winner is not the better edit, or the newer edit, or the one you meant to keep — it is the one whose bytes sort higher. The rule buys agreement, not correctness, and correctness is not on the menu for any rule that must produce one row out of two independent ones.

RELATION(TO:) AHEAD ANYWHERE? BEHIND ANYWHERE? .DOMINATES .DOMINATED .EQUAL .CONCURRENT CAUSALITY DECIDES → KEEP MINE → ADOPT THEIRS → DONE CONTENT DIFFERS? ? RESOLVEMERGE TIE-BREAK ADD-WINS ON PRESENCE LEX ON CANONICAL BYTES SAME WINNER ON EVERY DEVICE NOT THE BETTER EDIT — JUST THE HIGHER BYTES DETECTED IS NOT RESOLVED — AGREEMENT, NOT CORRECTNESS

A namespace can, though, choose an encoding that makes the arbitrary rule mean something: per-conversation read marks ride the settings namespace as fixed-width integer-seconds strings (ReadState.encode), so a lexicographic > agrees with a numeric one and the "larger string" tie-break picks the later watermark. What the tie-break is deliberately not is "larger updatedAt" — that would reintroduce the skew rule in precisely the case where it still had teeth.

5. The weakest rule that works

A version vector is not free: a bookkeeping row per synced row, a field on the wire, a permanent dimension per device. So the design pays for it only where the data genuinely needs it, and RowClock.causalEntities is the single table that says where — notebook, saga, plans, and contacts (with RelationshipState folded in beside Contact). It is deliberately the only such list, because a split between "which namespaces are causal" and "which entities map to them" would silently stop bumping a namespace's vectors: no error, only lost merges. Everything else gets a weaker rule, justified by the shape of its data.

Messages: monotone bits, and a fold. Delivery only ever goes from false to true, so the merge is if e.isDelivered, !existing.isDelivered { existing.isDelivered = true } — order-independent, idempotent, needing no history at all. The identity backfills (peerUserIDHash, senderName) are fill-if-empty and never overwrite a held value, so they can only gain information. Message content, however, is no longer immutable: edits and deletes exist. The recorded reason for not mutating the rows is that it would have forced vector clocks onto messages and lost the edit history, so instead a message became a fold over an append-only set of immutable ops, resolved by MessageFold.merge — a delete dominates by scope precedence (everyone > me > live) and purges the plaintext; otherwise edits are last-writer-wins on (editedAt, editOpID). That last rule has a wall clock in it; hold that thought for §8.

Memory: insert-if-absent. The memory log is append-only, so applyMemory is one line of logic — if this id is not present, insert it — with no vector, no tombstone, and no comparison. It is the weakest rule in the system, commutative and idempotent for free.

The precondition is doing real work, and there is a scar to prove it. Calendar events were once ingested into the memory log by deleting a time window and reinserting it with fresh ids on every pass. Under insert-if-absent, siblings dutifully resurrected the rows the ingester had just deleted and every device minted new ids each cycle: unbounded duplication, and a memory root that never converged. The fix was not a stronger merge rule but the recognition that these rows were never append-only — each device re-derives them from EventKit — so they left sync entirely, which applyMemory still enforces against stale peers that ship them.

A contact's device roster: single-writer, so not causal — on purpose. peerDevices is mutable and does sync between your devices, and it deliberately merges last-write-wins on the contact's own lastAdvertisedAt. The roster has exactly one author, the contact, whose advertisements are already totally ordered by their own clock — and causal add-wins would resurrect a device the owner deliberately dropped the instant a stale advertisement raced the drop. The stronger, more general machinery would have been the less safe choice.

6. Finding a local edit without a hook at every write site

Something has to call bump. The obvious place is every write site — and counting them is the argument against it: notebook edits from the UI, the agent's distillation writer on its own context, settings screens, contact import, plan updates, the reconcile applier itself. A missed call is not a crash; it is a row whose vector never advances and therefore loses every future comparison — the same silent, unfalsifiable loss this article opened by removing.

So the bump is not at the write sites. It is at the one place every write passes through: a single observer on ModelContext.didSave, registered with object: nil so it also catches saves committed on the background agent runner's own context. RowClock.observe fetches the live rows and the RowVersion entries for the affected namespaces and bumps any row whose updatedAt differs from the syncedAt we last accounted for.

updatedAt is still being read — but look at how. It is a change detector, never an ordering. The comparison is e.syncedAt != updatedAt, between the row's current value and one this same device recorded from this same clock, and only inequality is tested, never <. Skew cannot reach it: a device with a wildly wrong clock still notices that a value changed.

Two details keep the loop from feeding itself. On the apply side, RowClock.record stores the post-merge syncedAt, so a row just adopted from a sibling is not misread as a fresh local edit on the next save. And on the registration side, a namespace with no RowVersion entries at all is being seen for the first time, so its existing rows register with an empty vector rather than a bumped one. The reason is a one-line derivation: an empty vector is dominated by every non-empty one, so "I have no history" resolves to "adopt whatever the fleet has" — exactly the claim a device that merely launched is entitled to make. The same rule runs after a cloud-anchor recovery, so a device restored from an old backup cannot out-version the fleet.

7. Why the clock stays out of the digest

From Reconciling by Digest: each namespace hashes into a Merkle root over per-row content leaves, and equal roots mean the two devices are provably in sync. It is tempting to fold the vector into the leaf, so equal roots would mean "same content and same history". That sounds strictly better. It is self-defeating, in three steps.

Content-equal with vectors unequal is the normal steady state, not an anomaly. The pull is one-directional and each end runs it independently, so the instant A merges B's row, A's vector is the elementwise max and B's is still its own — identical bytes, different histories, and B will not catch up until its own pull fires. A digest committing to the vector calls that divergence while the data is identical.

The repair for the pseudo-divergence manufactures more of it. resolveMerge returns the merged vector for every outcome, including .keep. So "nothing about my content changed" would still move my leaf, change my root, trigger a re-advertise, and pull a bucket of rows out of my sibling — whose own merge then moves its root. The clock's bookkeeping would have become the sync traffic.

And some vector divergence is genuinely durable. One recorded case: a version of respond that scoped shipped tombstones to the mismatched key ranges withheld the tombstones of keys deleted on both devices, leaving their vectors permanently divergent. Recoverable, precisely because vectors are not hashed — had they been, the namespace would have advertised a mismatch no transfer could close.

So RowVersion is the one @Model that backs no digest leaf — bookkeepingEntityName is derived from the type rather than written as a string, so a rename cannot silently desync it — and affectsDigest returns false for a save that touched only RowVersion, so the clock's own bump saves schedule no advertisement at all. The set of entities that do back a leaf is an allowlist rather than a blocklist, learned the hard way: under an earlier "anything but RowVersion dirties the state" rule, sealing a digest advertisement rewrote the libsignal ratchet store, which dirtied the state, which scheduled another advertisement — a loop between siblings that never quiesced.

The rule underneath all of it is one sentence. The digest answers do we hold the same content?; the vector answers how did we come to hold it? Only the first is a fleet-wide fact — the second is per-device by nature, the same family as a message's local observedAt and its per-device peerPubkey, both excluded from the message leaf for exactly this reason.

8. What this does not give you

Dimensions are forever, and they grow with devices, not with time. A vector entry is a 64-character hex device key mapped to an integer, stored on every row that device has ever edited, and nothing in the code prunes one. It is not even bounded by the devices you currently own: a device removed from the account re-onboards with a fresh identity, so a phone removed and re-added is two dimensions in every row it ever touched.

Pruning is harder than it looks, which is why it does not exist. If A drops a departed device's counter from a row's vector and B has not, a pair that previously compared .dominated can come back .concurrent, flipping a merge from "adopt" to "tie-break" — silently, toward the wrong answer. Safe pruning needs every device to drop the same dimension: the fleet-wide unanimity problem the convergence barrier builds for tombstones, and it is not wired to vectors.

Concurrent edits still lose one, and lose it silently. Vectors tell you a conflict is real; they do not repair it. The unit of conflict is the whole row, so two devices editing different sentences of the same note produce one surviving note, not a merged one — and the code knows the difference (relation returned .concurrent, the tie-break ran) without surfacing it anywhere. The honest residual: the improvement over §1 is that the loss is deterministic and explicable rather than dependent on whose clock drifted, not that the loss is gone. Making it visible would mean a conflict surface; making it unnecessary would mean a per-field CRDT. Neither is built.

One wall clock survives, in messages. MessageFold.editIsNewer compares editedAt, which ComposeService.editMessage stamps from the editing device's Date() at whole-second resolution, tie-broken by a random 16-byte op id. That is last-write-wins on a wall clock, in the system that removed last-write-wins on a wall clock everywhere else. The exposure is bounded — only the author may edit their own message, so the race is between two of the author's own devices editing inside the skew window — but it is the §1 hazard, still standing, in the namespace with the most rows.

And the row is the granularity. A Contact and its RelationshipState share one clock, so a contact-only edit can causally dominate while carrying a stale relationship — hence the explicit guard against downgrading a newer local one. Splitting the relationship out is recorded as deferred: the guard is a patch over a granularity choice, and everyone knows it.

References & further reading