# Choosing Brains: the Model Bench

*An agent harness is, at bottom, a loop wrapped around a language model: gather context,
ask the model, act on the answer, repeat. Everything else in this series (planners,
skills, tool calls, huddles) rides on that one call. So the first real design question is
deceptively simple: **which model answers?** This article is about Pixie's answer: not one
model but a bench of them, one switch that decides who plays, and one rule that nothing
ever benches the whole team.*

This is the first deep dive of the series, sitting directly under [Agent](00-agent.md),
which explains why a relationship-care app ships an agent at all, and why its cognition
is local-first. Two Mechanism articles sit beneath this one:
[Mechanism: On-Device Curation](06-mechanism-on-device-curation.md) and
[Mechanism: External Routing & Fallback](07-mechanism-external-routing.md).

You do not need any machine learning here; we treat every model as a black box with a
`complete()` method. What we care about is the *harness*: how a model is chosen, what
happens when the chosen one is unavailable, and who gets to decide.

---

## 1. The tension: small and here vs. big and elsewhere

Every agent that runs on a phone faces the same fork.

**On-device models** are private by construction (the prompt never leaves the hardware)
and they are *always there*: no network, no account, no key, no server having a bad day.
But a model that fits in a phone's memory is small, and small models are weaker at
planning, at following long instructions, and at calling tools correctly.

**External models** are dramatically more capable. External here means one of two
things: a big open-weights model (one whose trained parameters are published, so anyone
can download and run it) on your own computer, or a frontier model (one of the biggest
commercial models) behind an API key, a personal credential that lets the app call
that provider on your account. But each comes with a prerequisite (a
running machine, a valid key) and a cost (your prompt travels somewhere else).

Most apps resolve this fork by picking a side. Pixie's stance, inherited from the
local-first trust decision in [Agent](00-agent.md), is different: the fork is the user's
to resolve, per device, at any time, and the app's job is to make every choice *safe to
make*. Concretely that means three things, and they are the spine of this article:

1. A **bench** of model sources, from an always-available floor up to
bring-your-own-frontier.
2. A **single routing truth**: one Settings switch that alone decides where cognition
runs.
3. **Graceful degradation**: every choice, when its prerequisite is missing, falls back
to on-device instead of failing.

> **Figure:** The user chooses among built-in, community, and external model seats while the built-in Apple system model remains the availability floor beneath them all

The bench has three seats. Let us take them in order of increasing ambition.

---

## 2. The floor: Apple's built-in model

The first seat is the one that is always occupied. Every iPhone that runs Pixie ships
with Apple's built-in system language model, exposed through the FoundationModels
framework (`SystemLanguageModel` driven through a `LanguageModelSession`). Pixie wraps it
in a type called `AppleFoundationProvider`, which implements the app's one model-facing
interface, `LLMProvider`. The whole interface amounts to: "take a list of messages and
some tool definitions, return a completion (or a stream of chunks)."

Every brain on the bench, local or remote, hides behind that same interface, which is what
makes swapping them a routing decision rather than a rewrite.

The built-in model matters for two reasons that have nothing to do with its raw quality:

- **It has no prerequisite.** Nothing to download, nothing to configure, no key to paste.
In Settings it is described exactly this way: *"The built-in on-device model. Private,
always available."*
- **It is the floor everything else lands on.** As we will see in section 6, every other
arm of the router names this provider as its fallback. When any prerequisite anywhere
is missing, the agent thinks *here*.

One honest wrinkle, because it shapes the deeper articles. The rest of the harness
passes around generic tool definitions: a name plus a JSON schema for the arguments
(the machine-readable list of what a call needs) handed to the model so it can ask for
a call. FoundationModels has its own native idea of a tool (a Swift `Tool` protocol
with typed, `@Generable` arguments), and that is a different shape.

`AppleFoundationProvider` does not pretend otherwise. Generic tools are dropped with a
debug log, and the provider attaches *native* FoundationModels tools instead: the Notebook
and the current date always, both being permission-free; a Calendar tool once Settings →
Agent → Context → Calendar is on; and the file tools once Context → Files is on and a
folder is actually connected.

How the harness gets reliable tool use out of small models anyway (by splitting the work
across cooperating sessions) is the subject of
[The L1–L2–L3 Harness](02-the-l1-l2-l3-harness.md).

---

## 3. Seat two: curated community models

The second seat is for users who want a *stronger* on-device brain and are willing to
spend a download and some gigabytes for it. Pixie can run open-weights models from the
`mlx-community` organization on Hugging Face, the public hub where open model weights
are published. These are models converted to run on Apple silicon via the MLX runtime,
and they run entirely on the phone. They flow through the same
`AppleFoundationProvider` seam: the provider builds its
session against the chosen MLX model when it is selected and downloaded, and against the
system model otherwise.

The interesting design problem here is not running the models; it is **choosing which
ones to offer**. The `mlx-community` org contains thousands of repositories, and most
would be a trap in an agent harness. Some are base checkpoints with no chat template:
raw pretrained weights that were never taught the shape of a conversational turn. Some
are text-only models that cannot read images. Some cannot call tools.

Pixie's catalog (`MLXModelHubCatalog`) fetches the org and parses it into a four-level tree,
`family (Qwen) → series (Qwen 2.5) → size (1.5B · Coder) → quant (4-bit)`,
where each leaf is one downloadable repo (the size is the parameter count, so 1.5B is
1.5 billion; the quant is how many bits each of those parameters is stored in).

The catalog then applies a curation bar: the browser only surfaces models that can do **vision and tool
calling**, because the skills-and-plan loop needs both to work the same way it does on
the Apple system model. A model that cannot call tools is not a smaller brain in this
harness; it is a dead end.

Whether a repo qualifies is itself a judgment call made by heuristics: Hugging Face
metadata has no reliable "can call tools" flag, so the catalog keys off
instruct/chat-tuned naming. The quantization tiers (4-bit, 8-bit, BF16) trade download
size against fidelity: fewer bits per parameter means a smaller download and a slightly
less faithful model. And the device itself gets a vote: a model can be marked
*"Needs an iPhone on iOS 27 with enough memory."*

All of that filtering machinery is its own story:
[Mechanism: On-Device Curation](06-mechanism-on-device-curation.md).

Two footnotes worth knowing. First, the curation bar is a *default*, not a cage: there is
an escape hatch to add any `mlx-community/<repo>` by name, on the user's own judgment.
Second, the built-in Apple model is deliberately *not* listed in this browser. It is its
own radio row, not one entry among thousands, because it plays a structurally different
role: it is the floor.

---

## 4. Seat three: External (a category, not a provider)

The third seat is where most apps would put "the cloud." Pixie instead puts a
**category**. The Settings row reads: *"Ollama on your Pixie Homebase, or your own
OpenAI / Claude API key."* The common thread is ownership. Every brain in this category
is something the *user* brings:

- **Your own Ollama runtime.** [The Homebase](04-the-homebase.md) (Pixie on your Mac,
a full peer of your phone) can run [Ollama](https://ollama.com), a local server for
large open-weights models. Pixie discovers it by probing the default loopback port,
the network address a machine uses to reach programs running on itself, with manual
entry for a non-default one. It also *tests before persisting*: a candidate
endpoint that fails its probe is reported, never stored. The Homebase advertises its
installed models to your other devices over the sealed sibling channel: the
end-to-end-encrypted relay lane between your own devices, which the server cannot read
(that channel belongs to the sibling
[GroupThink](/articles/groupthink/00-groupthink.md) series). Your handset can then route
inference requests to it. Even the pull list on the Homebase is curated the same way
the MLX browser is: a short list of tags known to be capable of tool calling, plus
free-text for anything else.
- **Your own OpenAI or Claude key.** Paste an API key and calls go straight from the
device to that provider: there is no Pixie server in the middle of the path. Keys
live in the device Keychain (iOS's encrypted credential store, which other apps cannot
read) and are displayed only obfuscated (`sk-proj•••••`). Each one has a *Test*
button that fires a real authenticated request billing zero tokens: a
`GET /v1/models`, where a 200 response (HTTP's "success" code) proves the key works.
You can hold a ring of keys per provider;
the first key in the ring is the active one.
- **A Homebase someone shared with you.** A friend can grant your account access to
*their* Homebase's models. On your side that pane is deliberately read-only: you
select a model with a checkmark, and nothing more; their machine, their management.
The grant-and-revoke machinery is
[Mechanism: Sharing a Homebase](11-mechanism-homebase-sharing.md).

Inside the External pane, a segmented dropdown picks which of these arms is live. That
dropdown is the reason External is one radio row rather than three: from the router's
point of view there is one question (*"is cognition leaving this device's on-device
models?"*) and one category answers it. Which arm, and how each is wired and probed, is
the second deep dive under this article:
[Mechanism: External Routing & Fallback](07-mechanism-external-routing.md).

---

## 5. One routing truth

Three sources, several arms, key rings, adverts, shared machines: this could easily
become a pile of interacting toggles where nobody can answer "so where does my prompt
actually go?" Pixie collapses it to one type and one switch.

The type is `LLMRouter`: the single point that decides which `LLMProvider` a call
site (any spot in the code that needs a model) actually gets. Chat, briefs, the
planner: none of them know about Ollama or keys or adverts; they ask
the router for a provider and get one. (Huddles are the one deliberate exception, and they do
not consult the router at all: huddle reasoning is constructed against the on-device provider
directly, so no configuration can route a friend's context to a cloud model. See
[The Agent in Your Day](05-the-agent-in-your-day.md).)

The switch is the **model-source radio** in Settings → Agent: three rows (Apple
Intelligence, Community, External) of which exactly one is selected, like the station
buttons on an old car radio. The router's
`resolve()` reads that radio first, and the radio alone decides the route:

| Radio | Prerequisite | Route |
| --- | --- | --- |
| Apple Intelligence | none | on-device, system model |
| Community | chosen model downloaded & ready | on-device, your MLX pick |
| External · Ollama | verified runtime, or a fresh Homebase advert (< 10 min old) | your own Ollama |
| External · OpenAI | a key in the Keychain | OpenAI, your key |
| External · Claude | a key in the Keychain | Anthropic, your key |

Any row whose prerequisite is missing routes on-device. That fallback is the subject of
its own section below.

Two disciplines keep this switch trustworthy.

**Checkmarks are configuration; the radio is routing.** Everything *inside* a configure
pane (picking an MLX quant, selecting an Ollama model, saving a key) sets a checkmark.
It never flips the radio. This is a deliberate consent boundary. Pasting an OpenAI key
should not silently start sending your conversations to OpenAI; it should make the
External row *ready*, and only your explicit tap on that row changes where cognition
runs. The inverse courtesy also holds. Selecting a source whose configuration is not
finished yet (Community with no model chosen, External that cannot route) auto-opens its
pane so you can finish. Until you do, the router simply resolves on-device.

> **Figure:** Saving and testing an external key makes the source ready, but only an explicit model-source radio selection changes the route away from on-device

**The decision is made per call, not at construction.** The router re-resolves every time
a call site asks for a provider. So a key you just added or a radio you just flipped
applies on the very next send: no restart, no rebuilt view models, no stale provider
captured in a closure somewhere. (The router even honors a retired setting: an old
install whose radio still says the short-lived value `"openai"` keeps routing correctly,
mapped into the External category. Stored user intent outlives UI redesigns.)

---

## 6. The rule: every arm degrades to on-device

Now the principle the whole bench is built around: **the agent never fails to route.**
There is no state of the settings, the network, or the world in which asking for a
provider returns an error. Instead, every arm names its prerequisite, and a missing
prerequisite means one thing: fall back on-device.

Note that the ladder has two rungs even *after* the fallback. The router falls back to
the user's on-device catalog choice first. If the chosen MLX model is not downloaded, or
the device cannot run it, that provider in turn falls back to the built-in system model.
The
floor is genuinely the floor.

> **Figure:** A failed external source falls first to the user's local model choice and then, if that model is not ready, to the always-present Apple system model

The fallback also runs at two *times*, because failures do not politely wait for
`resolve()`:

- **At resolve time**, the router checks prerequisites before choosing a route. The
Ollama arm is the instructive case. A handset only routes to the Homebase while the
Homebase's model advert (the periodic "I'm awake, here are my models" announcement
it sends) is *fresh*. A Mac that has gone to sleep is therefore skipped
instantly instead of costing a doomed timeout on every send. Freshness turns "fail
slowly, every time" into "fall back instantly". The windows and the reasoning behind
them are in [Mechanism: External Routing & Fallback](07-mechanism-external-routing.md).
- **Inside the provider**, for the races that resolve-time checks cannot close: the advert
expired, or the shared Homebase was revoked, *between* the routing decision and the actual
call. A call site cannot re-route mid-call. So the two **relay-routed** providers (your
own Homebase, and a friend's) carry the on-device provider as an internal fallback, and
the race falls back rather than trapping. The arms that talk to a live endpoint directly
(a loopback runtime or a cloud API) have no such internal net. For those, a failure after
the routing decision surfaces as an error on the turn. (Even a corrupted stored value
degrades safely: an unreadable "whose Homebase" record is treated as "my own" rather than
stranding routing.)

Why insist on this? Because of what the agent is *for*. Pixie's agent nudges you before
a friend's birthday, summarizes a huddle, drafts the message you asked for. An error
modal ("configure a model to continue") at any of those moments converts a relationship
tool into an IT ticket. A smaller answer beats no answer, always.

And the tradeoff, stated honestly: graceful degradation means capability can drop
*without the current operation failing*. Your carefully chosen 70B model (70 billion
parameters) naps with the
Mac, and the next reply quietly comes from a phone-sized brain.

A downgrade that was silent *and* invisible would be a dark pattern, so the state is
surfaced where you would look
for it: the Model
section in Settings leads with a live banner naming the *resolved* route ("External ·
qwen2.5:7b", "On-device · Apple Intelligence"), and the External panes flag a stale
Homebase in orange: *"Not heard from recently — inference falls back on-device until
it's back."*

The system never lies about where it is thinking; it just refuses to stop thinking while
you fix it.

---

## 7. Where the deep dives go

This article gave you the bench and the switch. Two mechanism articles sit directly
below it:

- [Mechanism: On-Device Curation](06-mechanism-on-device-curation.md): how the catalog
decides which `mlx-community` models qualify (the tool-calling and vision gates, quant
tiers, device capability checks), and why the built-in Apple model is always the floor.
- [Mechanism: External Routing & Fallback](07-mechanism-external-routing.md): Ollama on
the Homebase, the cloud key rings and their zero-token probes, the `LLMRouter` seam in
detail, and the full fallback ladder including the mid-call races.

Sideways and up: [The Homebase](04-the-homebase.md) is the machine that makes the Ollama
arm interesting, and [Mechanism: Inference over the Relay](10-mechanism-relay-inference.md)
is how a handset's prompt reaches it without any server reading it. And the question this
article deliberately postponed (*how can the small on-device floor run a real agent at
all?*) is exactly where [The L1–L2–L3 Harness](02-the-l1-l2-l3-harness.md) picks up.

---

## References

- Apple, [FoundationModels framework](https://developer.apple.com/documentation/foundationmodels):
`SystemLanguageModel`, `LanguageModelSession`, and the native `Tool` protocol.
- Hugging Face, [mlx-community](https://huggingface.co/mlx-community): the open-weights
model organization the Community catalog is built from; and
[MLX](https://github.com/ml-explore/mlx), the Apple-silicon array framework those
models run on.
- [Ollama](https://ollama.com): the local model runtime the External category's
self-hosted arm connects to.

*Next: [The L1–L2–L3 Harness](02-the-l1-l2-l3-harness.md).*
