Agent
Mechanism: Inference over the Relay
15 min read
Your phone holds the conversation; your Mac holds the big model — reaching it is the mechanism we call BigBrain. This article is the wire between them — how a handset asks its own Homebase to run a completion, over the same sealed channel the account's devices already use to sync, so the relay in the middle carries the question and the answer without ever learning that either one existed.
This is a mechanism deep-dive under The Homebase, which explains why a Mac in your account acts as a full peer at all. Its routing prerequisites — when the app decides to try this path, and the rule that every failure lands on-device — belong to Mechanism: External Routing & Fallback. Here we take the round itself: the advert that makes a phone aware of its Mac's models, the five-frame request cycle, the consistency gate that makes a separate context cache unnecessary, and the timeout ladder that keeps a sleeping desktop from ever costing you an answer.
1. The channel we refused to build
The obvious design is a new server feature. The phone POSTs a prompt to some
/v1/infer endpoint, the server holds it, the Mac polls for jobs, uploads the reply,
the phone polls for results. Every piece is familiar, and every piece is wrong for this
app: it hands the server a queue of prompts and replies — the exact content Pixie's
whole architecture exists to keep out of servers.
So remote inference adds zero new server surface. It rides the sealed sibling
channel: the end-to-end-encrypted frames the devices of one account already exchange
to keep their local databases in sync. On the wire these are sync_delta messages —
each one encrypted on the sending device for one specific sibling device, relayed as
opaque ciphertext, and decrypted only at the far end. The relay stores and forwards;
it cannot read.
Inference reuses that envelope without even minting a new outer message type. Inside the encrypted payload, a frame carries a small string kind, and the inference slice simply defines six new kinds (the same trick InstaFetch, the cross-device file-fetch feature, used before it):
ext_models_advert Homebase → all siblings "here is what my runtime can serve"
infer_request handset → Homebase "run this completion for me"
infer_accept Homebase → handset "model present, digest matches; starting"
infer_chunk Homebase → handset a batched slice of streamed text (advisory)
infer_done Homebase → handset terminal: status + AUTHORITATIVE full text
infer_cancel handset → Homebase "stop generating"
From the relay's perspective, a phone asking its Mac to think is indistinguishable from a phone syncing a note. That is the point.
One invariant governs every payload above, and it is written in capital letters in the
source: these frames carry no identities. No user hash, no public key, nothing that
names an account ever goes inside an inference payload. The sealed channel itself
already authenticates both ends — only a device of your account can decrypt the
frame — so the payload's only correlator is a random 16-byte request_id minted per
request. An identifier you never put on the wire is an identifier no bug, log, or
future refactor can leak.
2. The advert: how a phone learns its Mac can think
Before a handset can route a request, it has to know two things: which models the Homebase serves, and where to send the request. Both arrive in one broadcast frame.
The Homebase runs Ollama, a local model server, and Pixie's
ExternalRuntimeStore tracks its endpoint with a verify-before-trust discipline (see
Mechanism: External Routing & Fallback). Whenever
that runtime is (re)verified reachable, or its installed model set changes — a new
model pulled, the runtime forgotten — the Homebase broadcasts an ext_models_advert
to every sibling device: the list of model tags it can serve, plus a timestamp. On app
start it re-verifies the persisted endpoint in the background, and a success fires the
same advert. A forgotten runtime advertises an empty list, which un-advertises.
Notably, the advert is not piggybacked on the devices' periodic heartbeat. The heartbeat frame is two bytes; strapping a model list to it would bloat the single most frequent message in the system to serve the rarest change. Adverts are event-driven, and the freshness math below is designed around that.
On the handset, the last advert lands in ExternalModelsAdvertStore: the model list,
when it arrived, and — the routing-critical part — which sibling device sent it. That
sender's device key is where every infer_request will be addressed. The store mirrors
itself into UserDefaults (the iOS preferences store) so a relaunched app still shows
the model list immediately; freshness is always re-evaluated against the receive time,
so the mirror can never make a dead Homebase look alive.
The store feeds two consumers:
- The Settings picker. Settings → Agent lists the advertised tags, so choosing
"run my chats on
llama3.3:70b" is a normal model selection, not a network configuration exercise. A stale advert shows a hint rather than hiding the list. - The router gate.
LLMRouter— the one function in the app that decides "which model answers this call" — only routes a handset to the remote path if the advert is fresh: less than 10 minutes old and non-empty. A stale advert means "probably unreachable," so the router falls back before the doomed 10-second accept wait rather than after it; the reasoning behind the freshness discipline lives in Mechanism: External Routing & Fallback.
3. One round, five frames
With a fresh advert in hand, a completion becomes a five-frame conversation:
handset Homebase
│ │
│── infer_request ──────────────────────────►│
│ (request_id, model, inline_turns, │ gate 1: runtime alive?
│ base_digest, params) │ model in live /api/tags?
│ │ gate 2: base_digest == our digest?
│◄────────────────────────── infer_accept ───│
│ │ Ollama streams…
│◄──────────────────── infer_chunk (seq 0) ──│
│◄──────────────────── infer_chunk (seq 1) ──│
│◄──────────── infer_done (ok, full_text) ───│
│ │
│── infer_cancel ───────────────────────────►│ (only on timeout / user cancel)
The request inlines everything the model needs: the full list of conversation turns
(role + text), the model tag from the advert, the system prompt, and generation
parameters (max_tokens, temperature). A tools_json field exists in the wire shape
but is always the literal "[]" in v1 — tool calls never cross this wire. The agent
loop — tool dispatch, memory, files — stays on the handset by design (see
Mechanism: CallTool — One Tool to Bind Them), and a call
that carries tools routes straight to the on-device model instead. The field is there
so a later slice can let the desktop model emit tool calls without a wire change, not
because anything uses it today.
On the Homebase, RemoteInferenceHost runs two gates before committing to generate:
Gate 1 — can I actually serve this? The stored runtime endpoint must be verified,
and the requested model must appear in a live fetch of Ollama's /api/tags (its
installed-models listing) — not the cached result of the last successful probe. The
distinction matters: a runtime that died since the last verification must answer
infer_done(model_unavailable) immediately, so the handset falls back on-device in one
round-trip, rather than accepting the job and stalling the requester into a 90-second
timeout.
Gate 2 — do we agree on the conversation? The request's base_digest must match
the Homebase's own digest of the same conversation. This is the consistency gate, and
it deserves its own section (next).
Both gates pass → the Homebase sends infer_accept and starts streaming from the local
Ollama runtime. Text deltas are batched — flushed into a seq-numbered infer_chunk
roughly every 250 ms or 512 bytes, whichever comes first — instead of forwarded
token-by-token. The constraint is relay ack pressure: every frame is a store-forward-ack
cycle through the relay, so a reply should be a handful of frames, not hundreds. The
batch bounds are tuned to still feel like a live stream on the handset's screen.
When generation completes, the terminal frame is infer_done(ok, full_text) — the
complete reply, regardless of what the chunks carried. And two small host-side rules
close the loop: a redelivered request (the sync channel's outbox retries frames whose
acks got lost) arriving while the original is still being served is dropped as a
duplicate, keyed by request_id; and an infer_cancel cancels the serving task
mid-stream, after which no terminal frame is owed — the requester already gave up.
4. The digest: why the Homebase needs no cache
Here is the design problem that shapes this mechanism most. A remote model needs context — the conversation so far. Where does the Homebase get it?
When a friend shares their Homebase with you, their machine has never seen your
conversations, so that protocol maintains an encrypted conversation cache on the host,
keyed so only the guest can read it (see
Mechanism: Sharing a Homebase). The same-account
case needs none of that, because of a fact the app already paid for: your devices
sync. The Homebase is a full sibling; its local database already holds a replica of
the handset's agent-chat memory, delivered by the same sync_delta machinery this
wire rides on. The synced replica is the context store. No second cache, no second
consistency regime, no second thing to encrypt and expire.
But replicas converge eventually, not instantly — the handset's newest turns might still be in flight when the request arrives. So the request carries a proof of agreement: the conversation digest.
InferWire.conversationDigest is a pure function both ends run over their own local
replica. It collects the conversation's memory rows — Pixie's agent memory is
append-only MemoryEvent rows: chat turns, tool pills, fact-update cards (see
Mechanism: Memory & the Notebook) — hashes each row
into a fixed-format leaf (its id, kind, body, and conversation id, under a domain
separator, in a canonical byte encoding), sorts the leaves, and folds them into a
single 32-byte Merkle root — a hash-tree fingerprint where equal roots mean equal
row sets. Sorting before folding makes the digest independent of insertion order and
storage backend; two databases that hold the same rows produce the same root, full
stop.
Two subtleties make this actually work:
The exclusion rule. The turns inlined in the request are the very rows most likely
not to have synced yet — the user's message was persisted moments before the request
went out. If the digest committed to them, every request would diverge. So both sides
exclude the inlined turns from their computation — and they exclude by matching
(role, body), not by row id, because the wire's inline turns carry no ids. Matching
on content removes the same logical rows from both computations regardless of which
side has persisted them yet. (If a body appears twice, every matching row is excluded —
on both sides equally, so the roots still agree.)
Recovering the conversation id. The LLMProvider seam — the small protocol every
model behind LLMRouter implements — deliberately carries no thread identifier, so the
request context is recovered from the store: the chat view model persists the user's
turn before invoking the provider, so the newest stored user-message row whose body
matches the request's last user turn names the thread. Call sites outside any
conversation (thread naming, distillation passes) resolve to nil — and both ends then
digest the same nil-thread row set, which stays consistent too.
Equal roots tell the Homebase: my synced view of this thread matches the handset's, so the inlined prompt is the only context in flight. It is not generating against a stale or forked view of the conversation, and the reply it produces belongs to a thread both replicas agree on. It is also the seam that lets a future slice stop inlining the whole history and lean on the replica for older turns — the gate that makes that safe is already on every request.
5. Divergence and recovery
When gate 2 fails, the Homebase does not guess. It answers infer_done(divergent) and
generates nothing.
The handset's recovery leans, again, on machinery that already exists. A divergent
reply means the two replicas are out of sync — which is precisely the condition the
sibling-sync engine repairs all day long. So the provider calls the broker's
markDivergent hook, which is wired to markStateDirty(): the standard "advertise my
state digests, let siblings pull what they're missing" reconcile cycle. The inference
wire gets no private resync protocol; it kicks the existing one.
Then it waits a grace period — 3 seconds by default, sized as the sync engine's 400 ms
digest debounce plus a reconcile round-trip — recomputes the digest fresh, and retries
exactly once. A second divergent lands on the on-device fallback like any other
failure. One retry is the honest number: divergence should be rare and transient (a
race with a frame in flight), and a retry loop would trade a slightly smaller answer
for a conversation that hangs — the wrong trade for a chat.
6. The watchdog ladder: 10 s, 90 s, 30 s
The requester side is orchestrated by RemoteInferenceBroker, the rendezvous between
an in-flight request and the inbound frames the sync inbox feeds it. It keeps a map of
pending requests by request_id, and for each one it arms a watchdog ladder —
three timers, each matched to a distinct failure mode with its own honest deadline:
phase timer what silence means here
───────────── ───── ──────────────────────────────────────────────
accept 10 s the Homebase is unreachable or asleep — its
gates are cheap (a tags fetch + a digest), so
ten silent seconds means nobody is home
first output 90 s accepted, but nothing generated yet — a big
model may be cold-loading into memory, and
that is slow but legitimate; give it room
mid-stream 30 s text was flowing, then stopped — the runtime
or the machine died with the job half done
The accept timer is armed the moment the request is registered; an infer_accept
re-arms the ladder at 90 s; every infer_chunk re-arms it at 30 s; infer_done
cancels it. When any timer fires, the broker finishes the request's event stream with a
typed timeout error and best-effort sends infer_cancel so the Homebase stops burning
cycles on an answer nobody is waiting for. Frames that arrive after a timeout find no
pending entry under their request_id and are silently ignored — a late reply from a
Mac that woke up too slowly is normal, not an error.
And then the rule that makes the whole feature safe to depend on: every failure —
no fresh advert, send failure, any timeout phase, model_unavailable, a host error, an
interrupted stream, the second divergence — is answered by silently running the same
request on the injected on-device fallback (the built-in Apple model provider, in
production). The fallback lives inside RemoteInferenceProvider because it cannot
live anywhere else: by the time a view model holds a provider and has started a
completion, there is no seam to re-route mid-call. The user sees an answer; at most it
reads like a smaller model wrote it. The agent must never fail to answer just because
the desktop died — that discipline, and why silence is the right UX here, is argued in
full in Mechanism: External Routing & Fallback.
7. Chunks are advisory; done is authoritative
The transport under this wire is an outbox with retries: a frame whose ack is lost gets redelivered, and delivery order is not guaranteed. Streaming UIs are usually built on transports that promise ordered, exactly-once bytes (an HTTP response stream, a WebSocket). We had a choice: build reliable ordered streaming on top of the relay — a sequence-number protocol with gap detection, retransmit requests, reassembly timeouts — or sidestep the need for it.
We sidestepped, by splitting the roles a stream usually conflates:
infer_chunkframes are advisory. They exist so the handset's screen shows text appearing live. The broker feeds them through aChunkAssemblerthat orders by sequence number and drops duplicates (a redelivered frame is expected, not a bug), and its output only has to be good enough for a progress display.infer_done.full_textis authoritative. The terminal frame carries the complete reply, assembled on the Homebase from the actual generation. Whatever the chunks showed, this is the answer that gets returned, persisted, and synced.
A lost chunk therefore costs a moment of visual smoothness — never a corrupted answer.
That is why the chunk protocol can stay trivially simple: correctness does not depend
on it. (Today the provider's complete() awaits the done frame and the default
protocol stream() wraps it; a chunk-fed live-streaming override is a planned
fast-follow, and it is safe to add precisely because chunks were never load-bearing.)
One more piece of forward-compatibility hides in the done frame: its status travels
as a plain string, decoded into the known set (ok, divergent, model_unavailable,
error) on arrival. A future Homebase that emits a status this handset has never heard
of degrades to error — which means "fall back on-device" — instead of failing to
decode. Unknown futures map to the safe path.
8. What deliberately stayed out of v1
The slice is intentionally narrow, and the edges are worth naming:
- Tools never cross the wire. Covered above; the guard is the first line of the
provider's
complete(). - No remote model pulls. The advert lists only models already installed on the Homebase; asking the Homebase to download a new model stays an action you take at the Homebase. The wire kinds for remote pulls are sketched but out of this slice.
- No oversized-reply spill. The relay's body limits comfortably fit chat-sized
replies today; spilling a giant
full_textthrough the blob store is a TODO for the day replies outgrow frames. - Metrics count, but cannot quote. The Homebase's usage ledger records same-account requests too (as "you") so its dashboard shows the whole machine's load next to guests' — but the ledger's store type holds token counts only; there is structurally no field for text (see Mechanism: Sharing a Homebase).
One implementation note that pays for itself in tests: RemoteInferenceHost compiles
for iOS too, even though only the Mac Catalyst build ever wires one up (the platform
split — Catalyst hosts, everything else requests — lives in the wiring layer, not the
class). That keeps the model-unavailable and divergence gates exercisable from the iOS
simulator test suite, where the rest of the harness's tests already live.
Step back and the shape of the mechanism is easy to state. Remote inference added no server endpoint, no queue, no cache, and no identity to any payload. It is six frame kinds inside an envelope that already existed, a pure hash function both ends already agreed on the inputs to, and three timers. Everything hard — encryption, delivery, retry, replica convergence — was infrastructure the app had already built and paid for. The best privacy property of the design is the one you can verify by reading the frame shapes: there is nothing in them to leak.
References
- Ollama — the Homebase's local model runtime; its
/api/tagslisting backs the host's live model gate. - Wikipedia: Merkle tree — the hash-tree construction behind the conversation digest.
- Apple, UserDefaults — the advert store's relaunch mirror.