An agent harness is, at bottom, a loop wrapped around a language model: gather context, ask the model, act on the answer, repeat. Everything else in this series (planners, skills, tool calls, huddles) rides on that one call. So the first real design question is deceptively simple: which model answers? This article is about Pixie's answer: not one model but a bench of them, one switch that decides who plays, and one rule that nothing ever benches the whole team.
This is the first deep dive of the series, sitting directly under Agent, which explains why a relationship-care app ships an agent at all, and why its cognition is local-first. Two Mechanism articles sit beneath this one: Mechanism: On-Device Curation and Mechanism: External Routing & Fallback.
You do not need any machine learning here; we treat every model as a black box with a
complete() method. What we care about is the harness: how a model is chosen, what
happens when the chosen one is unavailable, and who gets to decide.
Every agent that runs on a phone faces the same fork.
On-device models are private by construction (the prompt never leaves the hardware) and they are always there: no network, no account, no key, no server having a bad day. But a model that fits in a phone's memory is small, and small models are weaker at planning, at following long instructions, and at calling tools correctly.
External models are dramatically more capable. External here means one of two things: a big open-weights model (one whose trained parameters are published, so anyone can download and run it) on your own computer, or a frontier model (one of the biggest commercial models) behind an API key, a personal credential that lets the app call that provider on your account. But each comes with a prerequisite (a running machine, a valid key) and a cost (your prompt travels somewhere else).
Most apps resolve this fork by picking a side. Pixie's stance, inherited from the local-first trust decision in Agent, is different: the fork is the user's to resolve, per device, at any time, and the app's job is to make every choice safe to make. Concretely that means three things, and they are the spine of this article:
The bench has three seats. Let us take them in order of increasing ambition.
The first seat is the one that is always occupied. Every iPhone that runs Pixie ships
with Apple's built-in system language model, exposed through the FoundationModels
framework (SystemLanguageModel driven through a LanguageModelSession). Pixie wraps it
in a type called AppleFoundationProvider, which implements the app's one model-facing
interface, LLMProvider. The whole interface amounts to: "take a list of messages and
some tool definitions, return a completion (or a stream of chunks)."
Every brain on the bench, local or remote, hides behind that same interface, which is what makes swapping them a routing decision rather than a rewrite.
The built-in model matters for two reasons that have nothing to do with its raw quality:
One honest wrinkle, because it shapes the deeper articles. The rest of the harness
passes around generic tool definitions: a name plus a JSON schema for the arguments
(the machine-readable list of what a call needs) handed to the model so it can ask for
a call. FoundationModels has its own native idea of a tool (a Swift Tool protocol
with typed, @Generable arguments), and that is a different shape.
AppleFoundationProvider does not pretend otherwise. Generic tools are dropped with a
debug log, and the provider attaches native FoundationModels tools instead: the Notebook
and the current date always, both being permission-free; a Calendar tool once Settings →
Agent → Context → Calendar is on; and the file tools once Context → Files is on and a
folder is actually connected.
How the harness gets reliable tool use out of small models anyway (by splitting the work across cooperating sessions) is the subject of The L1–L2–L3 Harness.
The second seat is for users who want a stronger on-device brain and are willing to
spend a download and some gigabytes for it. Pixie can run open-weights models from the
mlx-community organization on Hugging Face, the public hub where open model weights
are published. These are models converted to run on Apple silicon via the MLX runtime,
and they run entirely on the phone. They flow through the same
AppleFoundationProvider seam: the provider builds its
session against the chosen MLX model when it is selected and downloaded, and against the
system model otherwise.
The interesting design problem here is not running the models; it is choosing which
ones to offer. The mlx-community org contains thousands of repositories, and most
would be a trap in an agent harness. Some are base checkpoints with no chat template:
raw pretrained weights that were never taught the shape of a conversational turn. Some
are text-only models that cannot read images. Some cannot call tools.
Pixie's catalog (MLXModelHubCatalog) fetches the org and parses it into a four-level tree,
family (Qwen) → series (Qwen 2.5) → size (1.5B · Coder) → quant (4-bit),
where each leaf is one downloadable repo (the size is the parameter count, so 1.5B is
1.5 billion; the quant is how many bits each of those parameters is stored in).
The catalog then applies a curation bar: the browser only surfaces models that can do vision and tool calling, because the skills-and-plan loop needs both to work the same way it does on the Apple system model. A model that cannot call tools is not a smaller brain in this harness; it is a dead end.
Whether a repo qualifies is itself a judgment call made by heuristics: Hugging Face metadata has no reliable "can call tools" flag, so the catalog keys off instruct/chat-tuned naming. The quantization tiers (4-bit, 8-bit, BF16) trade download size against fidelity: fewer bits per parameter means a smaller download and a slightly less faithful model. And the device itself gets a vote: a model can be marked "Needs an iPhone on iOS 27 with enough memory."
All of that filtering machinery is its own story: Mechanism: On-Device Curation.
Two footnotes worth knowing. First, the curation bar is a default, not a cage: there is
an escape hatch to add any mlx-community/<repo> by name, on the user's own judgment.
Second, the built-in Apple model is deliberately not listed in this browser. It is its
own radio row, not one entry among thousands, because it plays a structurally different
role: it is the floor.
The third seat is where most apps would put "the cloud." Pixie instead puts a category. The Settings row reads: "Ollama on your Pixie Homebase, or your own OpenAI / Claude API key." The common thread is ownership. Every brain in this category is something the user brings:
sk-proj•••••). Each one has a Test
button that fires a real authenticated request billing zero tokens: a
GET /v1/models, where a 200 response (HTTP's "success" code) proves the key works.
You can hold a ring of keys per provider;
the first key in the ring is the active one.Inside the External pane, a segmented dropdown picks which of these arms is live. That dropdown is the reason External is one radio row rather than three: from the router's point of view there is one question ("is cognition leaving this device's on-device models?") and one category answers it. Which arm, and how each is wired and probed, is the second deep dive under this article: Mechanism: External Routing & Fallback.
Three sources, several arms, key rings, adverts, shared machines: this could easily become a pile of interacting toggles where nobody can answer "so where does my prompt actually go?" Pixie collapses it to one type and one switch.
The type is LLMRouter: the single point that decides which LLMProvider a call
site (any spot in the code that needs a model) actually gets. Chat, briefs, the
planner: none of them know about Ollama or keys or adverts; they ask
the router for a provider and get one. (Huddles are the one deliberate exception, and they do
not consult the router at all: huddle reasoning is constructed against the on-device provider
directly, so no configuration can route a friend's context to a cloud model. See
The Agent in Your Day.)
The switch is the model-source radio in Settings → Agent: three rows (Apple
Intelligence, Community, External) of which exactly one is selected, like the station
buttons on an old car radio. The router's
resolve() reads that radio first, and the radio alone decides the route:
| Radio | Prerequisite | Route |
|---|---|---|
| Apple Intelligence | none | on-device, system model |
| Community | chosen model downloaded & ready | on-device, your MLX pick |
| External · Ollama | verified runtime, or a fresh Homebase advert (< 10 min old) | your own Ollama |
| External · OpenAI | a key in the Keychain | OpenAI, your key |
| External · Claude | a key in the Keychain | Anthropic, your key |
Any row whose prerequisite is missing routes on-device. That fallback is the subject of its own section below.
Two disciplines keep this switch trustworthy.
Checkmarks are configuration; the radio is routing. Everything inside a configure pane (picking an MLX quant, selecting an Ollama model, saving a key) sets a checkmark. It never flips the radio. This is a deliberate consent boundary. Pasting an OpenAI key should not silently start sending your conversations to OpenAI; it should make the External row ready, and only your explicit tap on that row changes where cognition runs. The inverse courtesy also holds. Selecting a source whose configuration is not finished yet (Community with no model chosen, External that cannot route) auto-opens its pane so you can finish. Until you do, the router simply resolves on-device.
The decision is made per call, not at construction. The router re-resolves every time
a call site asks for a provider. So a key you just added or a radio you just flipped
applies on the very next send: no restart, no rebuilt view models, no stale provider
captured in a closure somewhere. (The router even honors a retired setting: an old
install whose radio still says the short-lived value "openai" keeps routing correctly,
mapped into the External category. Stored user intent outlives UI redesigns.)
Now the principle the whole bench is built around: the agent never fails to route. There is no state of the settings, the network, or the world in which asking for a provider returns an error. Instead, every arm names its prerequisite, and a missing prerequisite means one thing: fall back on-device.
Note that the ladder has two rungs even after the fallback. The router falls back to the user's on-device catalog choice first. If the chosen MLX model is not downloaded, or the device cannot run it, that provider in turn falls back to the built-in system model. The floor is genuinely the floor.
The fallback also runs at two times, because failures do not politely wait for
resolve():
Why insist on this? Because of what the agent is for. Pixie's agent nudges you before a friend's birthday, summarizes a huddle, drafts the message you asked for. An error modal ("configure a model to continue") at any of those moments converts a relationship tool into an IT ticket. A smaller answer beats no answer, always.
And the tradeoff, stated honestly: graceful degradation means capability can drop without the current operation failing. Your carefully chosen 70B model (70 billion parameters) naps with the Mac, and the next reply quietly comes from a phone-sized brain.
A downgrade that was silent and invisible would be a dark pattern, so the state is surfaced where you would look for it: the Model section in Settings leads with a live banner naming the resolved route ("External · qwen2.5:7b", "On-device · Apple Intelligence"), and the External panes flag a stale Homebase in orange: "Not heard from recently — inference falls back on-device until it's back."
The system never lies about where it is thinking; it just refuses to stop thinking while you fix it.
This article gave you the bench and the switch. Two mechanism articles sit directly below it:
mlx-community models qualify (the tool-calling and vision gates, quant
tiers, device capability checks), and why the built-in Apple model is always the floor.LLMRouter seam in
detail, and the full fallback ladder including the mid-call races.Sideways and up: The Homebase is the machine that makes the Ollama arm interesting, and Mechanism: Inference over the Relay is how a handset's prompt reaches it without any server reading it. And the question this article deliberately postponed (how can the small on-device floor run a real agent at all?) is exactly where The L1–L2–L3 Harness picks up.
SystemLanguageModel, LanguageModelSession, and the native Tool protocol.Next: The L1–L2–L3 Harness.