Your phone holds the conversation; your Mac holds the big model. Reaching it is the mechanism we call BigBrain. This article is the wire between them. It shows how a handset asks its own Homebase to run a completion (one full model reply to a prompt) over the same sealed (end-to-end encrypted) channel the account's devices already use to sync. The relay server in the middle carries the question and the answer without ever learning that either one existed.
This is a mechanism deep-dive under The Homebase, which explains why a Mac in your account acts as a full peer at all. Its routing prerequisites (when the app decides to try this path, and the rule that every failure lands on-device) belong to Mechanism: External Routing & Fallback.
Here we take the round itself: the advert that makes a phone aware of its Mac's models; the five-frame request cycle; the consistency gate (a pre-flight check that both machines agree on the conversation), which makes a separate context cache unnecessary; and the timeout ladder that keeps a sleeping desktop from ever costing you an answer.
The obvious design is a new server feature. The phone POSTs a prompt to some
/v1/infer endpoint, the server holds it, the Mac checks in for jobs and uploads the reply,
and the phone checks back for the result. Every piece is familiar, and every piece is wrong for this
app: it hands the server a queue of prompts and replies: the exact content Pixie's
whole architecture exists to keep out of servers.
So remote inference adds zero new server surface: no new endpoint, no new server code that could ever see this traffic. It rides the sealed sibling
channel: the end-to-end-encrypted frames (small messages only the two devices at the ends can read) that the devices of one account already exchange
to keep their local databases in sync. On the wire these are sync_delta messages.
Each one is encrypted on the sending device for one specific sibling device, relayed as
opaque ciphertext (scrambled bytes that reveal nothing), and decrypted only at the far end. The
relay stores and forwards; it cannot read.
Inference reuses that envelope without even minting a new outer message type. Inside the encrypted payload, every frame carries a small string called its kind, which tells the receiving device what the frame means. The inference feature simply defines six new kinds (the same trick InstaFetch, the cross-device file-fetch feature, used before it):
| Kind | Direction | Meaning |
|---|---|---|
ext_models_advert |
Homebase → all siblings | "here is what my runtime can serve" |
infer_request |
handset → Homebase | "run this completion for me" |
infer_accept |
Homebase → handset | "model present, digest matches; starting" |
infer_chunk |
Homebase → handset | a batched slice of streamed text (advisory) |
infer_done |
Homebase → handset | terminal: status + authoritative full text |
infer_cancel |
handset → Homebase | "stop generating" |
From the relay's perspective, a phone asking its Mac to think is indistinguishable from a phone syncing a note. That is the point.
One invariant (a rule that must hold everywhere, always) governs every payload above, and it is written in capital letters in the source: these frames carry no identities. No user hash, no public key, nothing that names an account ever goes inside an inference payload.
The sealed channel itself already authenticates both ends: only a device of your account can
decrypt the frame. So the payload's only correlator (the one label that ties replies back to requests) is a random 16-byte request_id, minted fresh for each
request. An identifier you never put on the wire is an identifier no bug, log, or future
refactor can leak.
Before a handset can route a request, it has to know two things: which models the Homebase serves, and where to send the request. Both arrive in one broadcast frame.
The Homebase runs Ollama, a local model server, and Pixie's
ExternalRuntimeStore tracks its endpoint (the local network address where Ollama answers) with a verify-before-trust discipline: never assume it is reachable, prove it (see
Mechanism: External Routing & Fallback).
Whenever that runtime is verified (or re-verified) reachable, or its installed model set changes (a new
model downloaded, the runtime forgotten), the Homebase broadcasts an ext_models_advert to every
sibling device. The advert is the list of model tags it can serve (an Ollama tag names one installed model and
size, like llama3.3:70b), plus a timestamp. On app start the Homebase re-verifies the saved endpoint
in the background, and a success fires the same advert. A forgotten runtime advertises an empty
list, which un-advertises.
Notably, the advert is not piggybacked on the devices' periodic heartbeat: the small frame siblings send on a timer to say they are still around. The heartbeat frame is two bytes; strapping a model list to it would bloat the single most frequent message in the system to serve the rarest change. Adverts are event-driven (sent only when something actually changes), and the freshness math below is designed around that.
On the handset, the last advert lands in ExternalModelsAdvertStore: the model list,
when it arrived, and (the routing-critical part) which sibling device sent it. That
sender's device key is where every infer_request will be addressed. The store mirrors
itself into UserDefaults (the iOS preferences store) so a relaunched app still shows
the model list immediately. Freshness is always re-evaluated against the receive time,
so the mirror can never make a dead Homebase look alive.
The store feeds two consumers:
llama3.3:70b" is a normal model selection, not a network
configuration exercise. A stale advert shows a hint rather than hiding the list.LLMRouter (the one function in the
app that decides "which model answers this call") only routes a handset to the
remote path if the advert is fresh: less than 10 minutes old and non-empty. A
stale advert means "probably unreachable," so the router falls back before the
doomed 10-second accept wait rather than after it. The reasoning behind the
freshness discipline lives in
Mechanism: External Routing & Fallback.With a fresh advert in hand, a completion becomes a five-frame conversation. The
handset sends infer_request: the request id, the model tag, the conversation turns written directly into the frame, a base
digest (a fingerprint of the shared history; section 4 unpacks it), and generation parameters. The Homebase answers infer_accept only after two cheap
gates: is the runtime alive with that model in its live /api/tags listing, and does
base_digest match its own? Then it streams infer_chunk frames and closes with
infer_done carrying the full text. The one frame the handset may send afterwards is
infer_cancel, on timeout or user cancel.
The request inlines everything the model needs: the full list of conversation turns (role +
text), the model tag from the advert, the system prompt (the standing instructions the model always sees), and generation parameters
(max_tokens, temperature: knobs for reply length and randomness). A tools_json field exists in the wire shape but is always the
literal "[]" in v1: tool calls never cross this wire.
The agent loop (tool dispatch, memory, files) stays on the handset by design (see Mechanism: CallTool, One Tool to Bind Them). A call that carries tools routes straight to the on-device model instead. The field is there so a later release can let the desktop model emit tool calls without a wire change, not because anything uses it today.
On the Homebase, RemoteInferenceHost runs two gates before committing to generate:
Gate 1: can I actually serve this? The stored runtime endpoint must be verified,
and the requested model must appear in a live fetch of Ollama's /api/tags (its
installed-models listing), not the cached result of the last successful probe. The
distinction matters. A runtime that died since the last verification must answer
infer_done(model_unavailable) immediately, so the handset falls back to its on-device model in one
round-trip, rather than accepting the job and stalling the requester into a 90-second
timeout.
Gate 2: do we agree on the conversation? The request's base_digest must match
the Homebase's own digest of the same conversation. This is the consistency gate, and
it deserves its own section (next).
Both gates pass → the Homebase sends infer_accept and starts streaming from the local
Ollama runtime. New text is batched (flushed into a sequence-numbered infer_chunk
roughly every 250 ms or 512 bytes, whichever comes first) instead of forwarded
token-by-token (a token is the word-sized fragment a model emits at each step). The constraint is acknowledgment pressure at the relay: every frame must be stored, forwarded, and confirmed received ("acked")
through the relay, so a reply should be a handful of frames, not hundreds. The
batch bounds are tuned to still feel like a live stream on the handset's screen.
When generation completes, the terminal frame is infer_done(ok, full_text): the
complete reply, regardless of what the chunks carried. Two small host-side rules
close the loop. First, a redelivered request (the sync channel's outbox retries any frame whose
acknowledgment got lost) arriving while the original is still being served is dropped as a
duplicate, keyed by request_id. Second, an infer_cancel cancels the serving task
mid-stream, after which no terminal frame is owed: the requester already gave up.
Here is the design problem that shapes this mechanism most. A remote model needs context: the conversation so far. Where does the Homebase get it?
When a friend shares their Homebase with you, their machine has never seen your conversations, so that protocol maintains an encrypted conversation cache on the host, encrypted under a key only the guest holds (see Mechanism: Sharing a Homebase).
The same-account case needs none of that, because of a fact the app already paid for: your
devices sync. The Homebase is a full sibling; its local
database already holds a replica (its own complete copy) of the handset's agent-chat memory, delivered by the same
sync_delta machinery this wire rides on. The synced replica is the context store. No second
cache, no second consistency regime, no second thing to encrypt and expire.
But replicas converge eventually, not instantly: the handset's newest turns might still be in flight when the request arrives. So the request carries a proof of agreement: the conversation digest.
InferWire.conversationDigest is a pure function (same input, same output, every time, on any machine) that both ends run over their own local replica.
It collects the conversation's memory rows. Pixie's agent memory is append-only MemoryEvent
rows (new rows get added, existing rows are never rewritten): chat turns, tool pills, fact-update cards (see Mechanism: Memory & the
Notebook). It then hashes each row (boils it down to a short, fixed-size fingerprint) as a fixed-format leaf: its
id, kind, body, and conversation id, in a canonical byte encoding (one agreed byte layout, so both machines hash identical input), under a domain separator (a
fixed label mixed into every one of these hashes so they can never collide with a hash the app
computes for some other purpose).
It then sorts the leaves and folds them into a single 32-byte Merkle root: a hash-tree fingerprint, built by repeatedly hashing the leaf hashes together, where equal roots mean equal row sets.
Sorting before folding makes the digest independent of insertion order and storage backend; two databases that hold the same rows produce the same root, full stop.
Two subtleties make this actually work:
The exclusion rule. The turns inlined in the request are the very rows most likely
not to have synced yet: the user's message was saved only moments before the request
went out. If the digest committed to them, every request would come up divergent. So both sides
exclude the inlined turns from their computation. They exclude by matching
(role, body), not by row id, because the wire's inline turns carry no ids. Matching
on content removes the same logical rows from both computations regardless of which
side has persisted them yet. (If a body appears twice, every matching row is excluded,
on both sides equally, so the roots still agree.)
Recovering the conversation id. The LLMProvider seam
(the small protocol every model behind LLMRouter implements) deliberately carries no
thread identifier, so the request context is recovered from the store. The chat view
model saves the user's turn before invoking the provider, so the newest stored
user-message row whose body matches the request's last user turn names the thread. Call
sites outside any conversation (thread naming, distillation passes) resolve to nil,
and both ends then
digest the same nil-thread row set, which stays consistent too.
Equal roots tell the Homebase: my synced view of this thread matches the handset's, so the inlined prompt is the only context in flight. It is not generating against a stale or forked view of the conversation, and the reply it produces belongs to a thread both replicas agree on. The digest is also the opening that lets a future release stop inlining the whole history and lean on the replica for older turns: the gate that makes that safe is already on every request.
When gate 2 fails, the Homebase does not guess. It answers infer_done(divergent) and
generates nothing.
The handset's recovery leans, again, on machinery that already exists. A divergent
reply means the two replicas are out of sync, which is precisely the condition the
sibling-sync engine repairs all day long. So the provider calls the broker's
markDivergent hook (the broker being the requester-side coordinator that tracks in-flight requests), which is wired to markStateDirty(): the standard "advertise my
state digests, let siblings pull what they're missing"
reconcile cycle. The inference
wire gets no private resync protocol; it kicks the existing one.
Then it waits a grace period: 3 seconds by default. That number is the sync engine's 400 ms digest debounce (the short pause it waits after a write before re-advertising, so a burst of edits produces one advert instead of twenty) plus room for a reconcile round-trip. After the wait, it recomputes the digest fresh and retries exactly once.
A second divergent lands on the on-device fallback like any other failure. One retry is the
honest number: divergence should be rare and transient (a timing race with a frame still in flight), and a
retry loop would trade a slightly smaller answer for a conversation that hangs, the wrong trade
for a chat.
The requester side is orchestrated by RemoteInferenceBroker, the meeting point between
an in-flight request and the inbound frames the sync inbox feeds it. It keeps a map of
pending requests by request_id, and for each one it arms a watchdog ladder of
three timers, each matched to a distinct failure mode with its own honest deadline:
| Phase | Timer | What silence means here |
|---|---|---|
| accept | 10 s | the Homebase is unreachable or asleep: its gates are cheap (a tags fetch + a digest), so ten silent seconds means nobody is home |
| first output | 90 s | accepted, but nothing generated yet: a big model may be cold-loading into memory, and that is slow but legitimate; give it room |
| mid-stream | 30 s | text was flowing, then stopped: the runtime or the machine died with the job half done |
The accept timer is armed the moment the request is registered. An infer_accept re-arms the
ladder at 90 s, every infer_chunk re-arms it at 30 s, and infer_done cancels it. When any timer
fires, the broker finishes the request's event stream with a typed timeout error and, best-effort,
sends infer_cancel so the Homebase stops burning cycles on an answer nobody is waiting for.
Frames that arrive after a timeout find no pending entry under their request_id and are
silently ignored: a late reply from a Mac that woke up too slowly is normal, not an error.
And then the rule that makes the whole feature safe to depend on: every failure (no fresh
advert, send failure, any timeout phase, model_unavailable, a host error, an interrupted
stream, the second divergence) is answered by silently running the same request on the injected
on-device fallback: the backup model the provider was handed when it was created (in production, the built-in Apple model provider).
The fallback lives inside RemoteInferenceProvider because it cannot live anywhere else: by
the time a view model holds a provider and has started a completion, there is nowhere left to
swap in a different route mid-call. The user sees an answer; at most it reads like a smaller model wrote it. The
agent must never fail to answer just because the desktop died. That discipline, and why silence
is the right user experience here, is argued in full in Mechanism: External Routing &
Fallback.
The transport under this wire is an outbox with retries: a frame whose acknowledgment is lost gets redelivered, and delivery order is not guaranteed. Streaming interfaces are usually built on transports that promise ordered, exactly-once bytes (an HTTP response stream, or a WebSocket: a persistent two-way connection). We had a choice: build reliable ordered streaming on top of the relay (a sequence-number protocol with gap detection, retransmit requests, reassembly timeouts) or sidestep the need for it.
We sidestepped, by splitting the roles a stream usually conflates:
infer_chunk frames are advisory. They exist so the handset's screen shows text
appearing live. The broker feeds them through a ChunkAssembler that orders by
sequence number and drops duplicates (a redelivered frame is expected, not a bug),
and its output only has to be good enough for a progress display.infer_done.full_text is authoritative. The terminal frame carries the complete
reply, assembled on the Homebase from the actual generation. Whatever the chunks
showed, this is the answer that gets returned, persisted, and synced.A lost chunk therefore costs a moment of visual smoothness, never a corrupted answer.
That is why the chunk protocol can stay trivially simple: correctness does not depend
on it. (Today the provider's complete() awaits the done frame and the default
protocol stream() wraps it; a chunk-fed live-streaming override is a planned
follow-up, and it is safe to add precisely because chunks were never load-bearing.)
One more piece of forward-compatibility hides in the done frame: its status travels
as a plain string, decoded into the known set (ok, divergent, model_unavailable,
error) on arrival. A future Homebase that emits a status this handset has never heard
of degrades to error (which means "fall back on-device") instead of failing to
decode. Unknown futures map to the safe path.
This first version is intentionally narrow, and the edges are worth naming:
complete().full_text through the blob store (the relay's side lane for large files) is a TODO for the
day replies outgrow frames.One implementation note that pays for itself in tests: RemoteInferenceHost compiles
for iOS too, even though only the Mac Catalyst build (Pixie
compiled for the desktop from the same codebase) ever wires one up. The platform
split (Catalyst hosts, everything else requests) lives in the wiring layer, not the
class. That keeps the model-unavailable and divergence gates exercisable from the iOS
simulator test suite, where the rest of the harness's tests already live.
Step back and the shape of the mechanism is easy to state. Remote inference added no server endpoint, no queue, no cache, and no identity to any payload. It is six frame kinds inside an envelope that already existed, a pure hash function both ends already agreed on the inputs to, and three timers.
Everything hard (encryption, delivery, retry, replica convergence) was infrastructure the app had already built and paid for. The best privacy property of the design is the one you can verify by reading the frame shapes: there is nothing in them to leak.
/api/tags listing backs the host's live model gate.