The Settings model browser looks like a boring list: a few names, some sizes, a download button. Behind it is a pipeline that pulls thousands of community model repos (repositories, the online folders where models are published) off the internet and throws most of them away on purpose. And it guarantees one strange invariant, a rule that must stay true no matter what: whatever the user downloads, deletes, mistypes, or cancels, the agent always has a brain. This article is how that list gets made, and why the built-in Apple model is the floor under everything.
This is a mechanism deep-dive under Choosing Brains: the Model Bench, which explains why Pixie runs its agent on-device at all and introduces the model-source radio: the pick-one selector in Settings that decides where the agent's brain comes from. Here we go underneath one arm of that radio: the on-device catalog, its capability gates, the download lifecycle, and the fallback story.
Pixie's on-device models come from mlx-community, an organization on Hugging Face
(the website where the machine-learning community publishes models). Its volunteers take
open-weight models (models whose learned numbers, their weights, are free for anyone to
download) and convert them into MLX format: Apple's framework for
running models efficiently on Apple-silicon chips, the processors in modern iPhones and Macs. It is a wonderful resource and a
terrible product surface. There are thousands of repos. The names are inconsistent
(Qwen2.5-1.5B-Instruct-4bit, gemma-3-4b-it-4bit, Llama-3.2-3B-Instruct-4bit: three
different naming grammars right there). Many of the models will not load on a phone. Many
more will load and then fail at the one thing Pixie needs them for.
That last point is the crux. Pixie's agent is not a chat toy. It plans, follows skills, and calls tools in a loop (see The L1–L2–L3 Harness). It also describes images from your files and conversations. When the user swaps the brain, none of that machinery is allowed to silently break. So the browser cannot be a mirror of the Hugging Face org; it has to be a curated view, built by answering three questions about every repo:
Everything in this article is one of those three questions, followed by what happens after the user taps Download, and what happens when any step of that goes wrong.
The catalog layer is a class called MLXModelHubCatalog. On first open (and on explicit
refresh) it queries the Hugging Face API for the mlx-community org. It sends three queries in
parallel, one per pipeline tag, the metadata field in which a repo declares what task
it is built for: text-generation for plain language models,
image-text-to-text and any-to-any for the vision-capable ones. The results are merged,
and duplicates of the same repo id are dropped.
The three-tag fetch is a scar with a lesson in it. An earlier version filtered by the one
obvious tag, text-generation, and thereby silently dropped every vision model:
vision models live under different tags, and nothing errored. The org also hosts speech-recognition
and embedding models under yet other tags; those are skipped on purpose, because the
agent cannot use them as a brain.
Each query asks the API to expand three fields per model, and each one feeds a gate you will meet below:
safetensors: the weight file's own inventory, a count of parameters (the model's
learned numbers) broken down by datatype, which gives an exact download size;config: the model_type string, which names the architecture (the model's internal layout);pipeline_tag: the reliable "takes images" signal.Raw repo ids are then parsed (by best-effort pattern matching, because the names are
community-written) into a four-level tree:
family (Qwen) → series (Qwen 2.5) → size (1.5B · Coder) → quant (4-bit).
The quant level is the leaf: one leaf equals one repo id equals one downloadable model. The
parser (MLXRepoParser) matches against some three dozen known family names, trying the
longest and most specific first, so codellama wins over llama. It pulls a version number from just
after the family token and finds a size token like 1.5B or 8x7B (or a keyword like "mini").
Meaningful variants (Coder, Math, Vision, Thinking) are tagged into the size label.
Anything that does not parse lands in a family called "Other", and "Other" is dropped from the browser entirely. An earlier build kept it as a catch-all, which just meant a bucket of opaque repo ids the user could not make sense of. A row you cannot name is worse than no row.
The finished tree is cached to disk (a JSON file with a seven-day shelf life), so the browser is instant and works offline after the first load; a pull-to-refresh re-fetches.
An MLX repo is just weights plus a config file. Actually running it is another matter. The installed runtime (the mlx-swift language-model package the app links) must have a factory for that architecture: code that knows how to build the model's layer layout, the structure those weights are numbers for. Somebody has to have implemented that structure first.
The catalog carries a hard-coded allowlist, supportedModelTypes: about sixty model_type
strings, the union of the runtime's two factory registries: LLMs (large language models, which
work in text) and VLMs (vision-language models, which take images as well as text). Think
llama, qwen3, gemma3, phi3, … plus vision types
like qwen2_5_vl, smolvlm, pixtral.
A fetched model whose config.model_type is not on the list is filtered out before the
tree is built. The reasoning is about where the failure would otherwise land: an
unsupported architecture does not fail in the browser, it fails at load time, after the
user has watched a multi-gigabyte download complete. Filtering early moves the failure to
the cheapest possible place: the model simply never appears.
One deliberate softness: a model whose config reports no model_type at all is kept.
Benefit of the doubt: an absent field is a metadata gap, not evidence of an exotic
architecture.
This is the pair of gates the whole browser hinges on. When the tree is built, a size node is only surfaced if it is both vision-capable and tool-capable:
// Only surface models the agent can actually DRIVE: multimodal (vision)
// AND tool-calling.
guard sz.isVision && sz.tools else { return nil }
Why such an aggressive AND? Because the bar is the built-in Apple model's job description. The
skills-and-tools plan loop has to work identically on a community model as it does on the system
model. That is what MLXTextExecutor's tool bridge exists for: the seam that gives a
downloaded model the same tool-calling interface the harness already speaks. (The loop itself is
the subject of Mechanism: CallTool, One Tool to Bind Them.)
And the file indexer and chat both hand the model images. A model that cannot call tools cannot execute a plan; a model that cannot see cannot describe your photos. Either way the user would have downloaded a dead end, and dead ends do not go in the browser.
The two capability signals are detected quite differently, and honestly.
Tool calling has no reliable per-model flag anywhere in Hugging Face metadata. What
actually determines it is whether the checkpoint (the saved copy of a model's weights) is
instruction-tuned, meaning trained a second time to follow conversational instructions. A base
(pretrained-only) checkpoint ships no chat template: the formatting layer that turns a
conversation plus a tool menu into the exact stream of tokens (the word-fragments a model
actually reads) it was tuned on. Without one, the model cannot
follow a tool-calling protocol at all. So the gate keys off the naming convention that
marks instruction tuning: the repo name contains instruct, chat, an -it suffix,
reasoning, or thinking. The code calls this what it is: "an availability hint, not a
guarantee." It is a heuristic, chosen because the alternative (downloading each model just to
probe it) is absurd. And a false positive fails gently: the model ignores the
tool menu, which the harness already tolerates.
Vision has a much better signal. The primary test is the pipeline_tag straight from
Hugging Face: image-text-to-text or any-to-any means the model takes images, full
stop. Behind it sits a fallback list of VLM-exclusive model_types for repos where the
API did not populate the tag, and that list deliberately omits the dual-use
architectures (gemma3, gemma4, mistral3, qwen3_5). Those names cover both
text-only and vision builds: the same model_type string loads as text in the LLM
factory and as vision in the VLM factory. The type alone would therefore mis-tag every text
build as vision; only the pipeline tag can tell them apart. Last of all comes a name
sniff (-vl-, vision) for repos with neither signal.
Put together, the funnel narrows from thousands of mlx-community repositories to the
browser's short list: the pipeline tag admits generative repos, supportedModelTypes
keeps what the runtime can load, the name parser keeps what it can present, and the
vision-and-tools gate keeps what the agent can drive.
Each leaf of the tree is a quantization tier: the same weights stored at different precisions. For the harness's purposes, quantization is simply bytes per weight: a "4-bit" build stores each parameter in roughly half a byte instead of the two bytes of the original BF16 (a 16-bit floating-point format). That trades a little quality for a download and memory footprint a phone can actually carry.
Within a size node the tiers are sorted numerically: 4-bit before 6-bit before 8-bit, with the float formats (BF16, FP16) at the end, so the sensible choice for a handset is the first one you see.
Every leaf shows a download size, and the catalog goes to some trouble to make that
number honest. When the Hugging Face API returns the safetensors expansion (a map from
datatype to parameter count), the size is computed exactly: parameters times bytes per
element (4 for F32, 2 for F16/BF16, 1 for the 8-bit types, and so on).
When the API does not populate that field, the catalog falls back to an estimate calibrated against real mlx builds, and the calibration is more interesting than you would guess:
| Tier | Bytes per parameter |
|---|---|
| 2-bit | 0.34 |
| 3-bit | 0.45 |
| 4-bit | 0.56 |
| 6-bit | 0.80 |
| 8-bit | 1.06 |
| BF16 / FP16 | 2.1 |
| FP32 | 4.1 |
Notice 4-bit is 0.56 bytes per parameter, not the naive 0.5. Real quantized checkpoints carry extra baggage: unquantized embeddings (the model's word-lookup tables, kept at full precision), per-block scale factors, and metadata. The constant is fitted so that, for example, a Qwen 1.5B at 4-bit estimates to about 837 MB, which is what the real download costs. A size label that is 12% short of the truth is the kind of small lie that erodes trust in every other number on the screen.
Alongside size, each curated model carries a minimum-RAM recommendation (minRAMGB,
6–8 GB for the shortlist below) used by the download prompt, a recommendation rather
than a hard wall, because peak memory depends on context length (how much conversation
the model is holding at once) and what else the phone is doing.
All of the above decides what is offered. One check decides whether MLX download is
offered at all: isDeviceCapable, true only on a real Apple-silicon device running iOS
27 or later. The simulator (the pretend iPhone developers run on a Mac) cannot execute
MLX kernels (the compiled GPU routines the
weights are actually run through), so there every MLX model reports a
terminal unsupported state, one it never leaves: the browser still renders, but nothing downloads.
The catalog also keeps a second, much smaller list beside the big tree:
OnDeviceModelCatalog.curated, a hand-vetted shortlist of four 4-bit models for one-tap
download: Qwen2.5 1.5B and Llama 3.2 3B for text, Qwen2-VL 2B and Gemma 3 4B for
vision, spanning roughly 1.1 to 3.2 GB. The full hub tree is for exploring; the shortlist
is for people who just want a good answer. Both funnel into the same download machinery.
Curation is a default, not a cage. The browser accepts a hand-typed repo id, and the
validation is deliberately minimal: isValidRepoID checks only that the string is a
plausible owner/name: exactly two non-empty parts, id-safe characters. No network
call confirms the repo exists; a bad id simply fails at download time with a clear error.
Added repos are persisted (as a small JSON list in settings) and listed beneath the curated
catalog; adding the same id twice stores it only once.
Two honest edges of this door. First, a custom repo bypasses the vision-and-tools gate: by typing an id yourself you are asserting you know what it is, and the fallback floor (section 10) catches you if you were wrong.
Second, a custom model's modality (text-only versus vision) is provisionally labeled text. The runtime auto-detects the real modality when the weights load, so the label is a placeholder, not a verdict. Removing a custom repo deletes its weights, and if it was the active model, the selection snaps back to the built-in Apple model: our first sighting of the floor.
DownloadStateOnce a model is chosen, OnDeviceModelStore (the lifecycle layer between the catalog
and the router) tracks it through a five-state machine:
enum DownloadState: Equatable {
case unsupported // not a capable device / OS for MLX
case absent // capable, not yet downloaded
case downloading(Double) // 0…1 progress
case ready // weights present (or the built-in system model)
case failed(String)
}
download() moves absent to downloading. Success lands ready; an error lands
failed. cancel returns a download to absent, and delete() walks a ready model
back to absent. unsupported is terminal: the hardware gate said no, and that answer
never changes.
States are seeded at app launch: on a capable device each MLX model starts ready or
absent depending on disk; on an incapable one, unsupported. The built-in Apple model
never enters the machine: asking for its state always returns ready.
A download runs inside a tracked task, keyed by model id, so it can be cancelled mid-flight. Under the hood the MLX runtime fetches the repo from the Hugging Face Hub with genuine 0-to-1 progress, then loads and caches the model container in memory: the same container later used for inference (actually running the model to produce answers), so the download doubles as a warm-up. Two small concurrency guards in this path (protections against work racing itself across threads) are worth naming, because each one is a bug that was lived rather than imagined:
progress(1.0) arriving after completion could overwrite ready
with downloading(1.0): a bar stuck at 100% forever. So progress updates apply only
while the state is still downloading.absent, and neither the
completion path nor the error path is allowed to resurrect it to ready or demote it
to failed. Cancel also removes partially written files, so the next attempt starts
clean.The subtlest choice is what "downloaded" means. The store does not scan a directory: it keeps
a persisted list of ready model ids, marked only after a load has actually succeeded. The reason
is ownership: the Hub client stores weights in its own cache
(Caches/huggingface/hub/models--<org>--<name>, plus a .locks sibling), a location the app
does not control, so a directory check against the app's own folder would see nothing.
The flip side is that deleting a model must chase the weights to where they really live:
delete() removes those Hub cache directories, evicts the in-memory container, and clears the
marker. All three, or "re-download" would be a lie that completes instantly against stale
files.
One deliberate ordering quirk: you may select a model before it finishes downloading. The browser shows it as your choice while the bar fills, and the agent, meanwhile, keeps answering. How? The floor.
A small but telling feature rides on top of this machinery: your model choice is synced across your devices. Picking a model calls a hook that fans the choice out to your other signed-in devices over the account's encrypted sibling-sync channel. The payload is just the model id plus a vision flag; the flag drives the chat composer's image-attach button.
Each sibling adopts the selection without echoing it back: the apply path is marked non-propagating, the standard trick for breaking sync loops (device A tells B, B tells A, forever). And if the selection is an MLX download the sibling does not have, it kicks off the download on its own. Your phone and your iPad converge on the same brain without either one asking.
Everything above can fail. The network can drop mid-download. A custom repo id can be a
typo. A device can be incapable. The user can delete the very model they had selected.
The reason none of this is scary is a single model with special status:
OnDeviceModelCatalog.appleSystem, the Apple Intelligence system model that ships with
the OS. Zero download bytes, zero RAM requirement, no state machine: it is defined as
present.
What makes it a floor rather than merely a default is that four independent layers all land on it:
AppleFoundationProvider (the provider that runs
on-device inference) checks at session-build time whether the chosen MLX model is
actually downloaded and loadable, and silently builds a system-model session when it
is not. The comment in the code states the contract plainly: fall back to the system
model so the agent always responds. This is what makes select-before-download safe.LLMRouter (the single point that decides which provider every agent
call site gets) constructs the on-device provider as the fallback arm of every
route: no OpenAI key, no Anthropic key, a stale advert from your
Homebase (an advert is the signed "here are my models"
announcement your Mac broadcasts),
a revoked shared
runtime. Each missing prerequisite resolves to on-device rather than to an error
(the full ladder is Mechanism: External Routing & Fallback).Why does an agent harness need this invariant so badly? Because an agent is a process, not a request. A chat app that fails one send shows a retry button. An agent that loses its model mid-plan strands a half-executed plan: tools called, state changed, no brain to continue. The router's promise that resolving a provider never fails is only honest because one model cannot be absent. Every other arm of the bench is allowed to be conditional precisely because this one is not.
The floor even covers the vision path where the hardware allows: on an iPhone 17 or newer (and on iPads and Macs), the built-in model accepts images too, so falling back does not necessarily mean going blind.
The browser's calm is manufactured, gate by gate. Three filters decide what you may see: a loadable architecture, a name the parser can present, and tool-calling and vision so the agent can drive it. Two numbers keep the offer honest: exact-or-calibrated sizes and a RAM recommendation. One hardware check decides whether downloads exist at all. A five-state machine with two concurrency guards governs the lifecycle, and one undeletable model backstops every failure path.
The moral, in the spirit of this series: curation here is not taste; it is an availability contract. Every row in the browser is a model that will load, that the harness can drive, and that the device can hold; and if any promise breaks anyway, the fallback is a model that was never allowed to be missing.
/api/models
endpoint, pipeline tags, and the safetensors metadata expansion.supportedModelTypes.