Agent

Mechanism: On-Device Curation

15 min read

The Settings model browser looks like a boring list: a few names, some sizes, a download button. Behind it is a pipeline that pulls thousands of community model repos off the internet, throws most of them away on purpose, and guarantees one strange invariant: no matter what the user downloads, deletes, mistypes, or cancels, the agent always has a brain. This article is how that list gets made — and why the built-in Apple model is the floor under everything.

This is a mechanism deep-dive under Choosing Brains: the Model Bench, which explains why Pixie runs its agent on-device at all and what the model-source radio is. Here we go underneath one arm of that radio: the on-device catalog, its capability gates, the download lifecycle, and the fallback story.

1. The problem: a firehose of repos and an agent with a job

Pixie's on-device models come from mlx-community, a Hugging Face organization where volunteers convert open-weight models into MLX format — Apple's array framework for running models efficiently on Apple-silicon chips. It is a wonderful resource and a terrible product surface. There are thousands of repos. The names are inconsistent (Qwen2.5-1.5B-Instruct-4bit, gemma-3-4b-it-4bit, Llama-3.2-3B-Instruct-4bit — three different naming grammars right there). Many of the models will not load on a phone. Many more will load and then fail at the one thing Pixie needs them for.

That last point is the crux. Pixie's agent is not a chat toy — it plans, follows skills, and calls tools in a loop (see The L1–L2–L3 Harness), and it describes images from your files and conversations. When the user swaps the brain, none of that machinery is allowed to silently break. So the browser cannot be a mirror of the Hugging Face org; it has to be a curated view, built by answering three questions about every repo:

Will the runtime even load it? (architecture check)
Can the agent drive it? (tool-calling and vision gates)
Can this device hold it? (size, quantization, and device capability)

Everything in this article is one of those three questions, followed by what happens after the user taps Download — and what happens when any step of that goes wrong.

2. From repo names to a tree

The catalog layer is a class called MLXModelHubCatalog. On first open (and on explicit refresh) it queries the Hugging Face API for the mlx-community org — three queries in parallel, one per pipeline tag: text-generation for plain language models, image-text-to-text and any-to-any for the vision-capable ones. The results are merged and de-duplicated by repo id.

The three-tag fetch is a scar with a lesson in it: an earlier version filtered by the one obvious tag, text-generation, and thereby silently dropped every vision model — they live under different tags, and nothing errored. The org also hosts speech-recognition and embedding models under yet other tags; those are skipped on purpose, because the agent cannot use them as a brain.

Each query asks the API to expand three fields per model, and each one feeds a gate you will meet below:

safetensors — per-datatype parameter counts, which give an exact download size;
config — the model_type string, which names the architecture;
pipeline_tag — the reliable "takes images" signal.

Raw repo ids are then parsed — heuristically, because the names are community-written — into a four-level tree:

family (Qwen)  →  series (Qwen 2.5)  →  size (1.5B · Coder)  →  quant (4-bit)

The quant level is the leaf: one leaf equals one repo id equals one downloadable model. The parser (MLXRepoParser) matches against some three dozen known family names (longest, most specific needles first, so codellama wins over llama), pulls a version number from just after the family token, finds a size token like 1.5B or 8x7B (or a keyword like "mini"), and tags meaningful variants — Coder, Math, Vision, Thinking — into the size label. Anything that does not parse lands in a family called "Other", and "Other" is dropped from the browser entirely. An earlier build kept it as a catch-all, which just meant a bucket of opaque repo ids the user could not make sense of. A row you cannot name is worse than no row.

The finished tree is cached to disk (a JSON file with a seven-day shelf life), so the browser is instant and works offline after the first load; a pull-to-refresh re-fetches.

3. Gate one: will the runtime even load it?

An MLX repo is just weights plus a config. Actually running it requires the installed runtime — the mlx-swift language-model package the app links — to have a factory for that architecture. The catalog carries a hard-coded allowlist, supportedModelTypes: about sixty model_type strings, the union of the runtime's LLM and VLM factory registries (llama, qwen3, gemma3, phi3, … plus vision types like qwen2_5_vl, smolvlm, pixtral).

A fetched model whose config.model_type is not on the list is filtered out before the tree is built. The reasoning is about where the failure would otherwise land: an unsupported architecture does not fail in the browser, it fails at load time — after the user has watched a multi-gigabyte download complete. Filtering early moves the failure to the cheapest possible place: the model simply never appears.

One deliberate softness: a model whose config reports no model_type at all is kept. Benefit of the doubt — an absent field is a metadata gap, not evidence of an exotic architecture.

4. Gates two and three: can the agent drive it?

This is the pair of gates the whole browser turns on. When the tree is built, a size node is only surfaced if it is both vision-capable and tool-capable:

// Only surface models the agent can actually DRIVE: multimodal (vision)
// AND tool-calling.
guard sz.isVision && sz.tools else { return nil }

Why such an aggressive AND? Because the bar is the built-in Apple model's job description. The skills-and-tools plan loop has to work identically on a community model as it does on the system model — that is what MLXTextExecutor's tool bridge exists for, the seam that gives a downloaded model the same tool-calling interface the harness already speaks (the loop itself is the subject of Mechanism: CallTool — One Tool to Bind Them). And the file indexer and chat both hand the model images. A model that cannot call tools cannot execute a plan; a model that cannot see cannot describe your photos. Either way the user would have downloaded a dead end, and dead ends do not go in the browser.

The two capability signals are detected quite differently, and honestly.

Tool calling has no reliable per-model flag anywhere in Hugging Face metadata. What actually determines it is whether the checkpoint is instruction-tuned: a base (pretrained-only) checkpoint ships no chat template — the formatting layer that turns a conversation plus a tool menu into the token stream the model was tuned on — so it cannot follow a tool-calling protocol at all. So the gate keys off the naming convention that marks instruction tuning: the repo name contains instruct, chat, an -it suffix, reasoning, or thinking. The code calls this what it is — "an availability hint, not a guarantee." It is a heuristic, chosen because the alternative (downloading each model to probe it) is absurd, and the failure mode of a false positive is a model that ignores the tool menu, which the harness already tolerates.

Vision has a much better signal. The primary test is the pipeline_tag straight from Hugging Face: image-text-to-text or any-to-any means the model takes images, full stop. Behind it sits a fallback list of VLM-exclusive model_types for repos where the API did not populate the tag — and that list deliberately omits the dual-use architectures (gemma3, gemma4, mistral3, qwen3_5). Those names cover both text-only and vision builds — the same model_type string loads as text in the LLM factory and as vision in the VLM factory — so the type alone would mis-tag every text build as vision; only the pipeline tag can tell them apart. Last of all comes a name sniff (-vl-, vision) for repos with neither signal.

Put together, the funnel looks like this:

mlx-community on Hugging Face         (thousands of repos)
      │  pipeline_tag ∈ {text-generation, image-text-to-text, any-to-any}
      ▼
fetched candidates
      │  config.model_type ∈ supportedModelTypes     — the runtime can load it
      ▼
loadable models
      │  parsed into family → series → size → quant  — "Other" dropped
      ▼
nameable models
      │  isVision AND supportsTools                  — the agent can drive it
      ▼
the browser

5. Quant tiers, and telling the truth about size

Each leaf of the tree is a quantization tier — the same weights stored at different precisions. For the harness's purposes, quantization is simply bytes per weight: a "4-bit" build stores each parameter in roughly half a byte instead of the two bytes of the original BF16, trading a little quality for a download and memory footprint a phone can actually carry. Within a size node the tiers are sorted numerically — 4-bit before 6-bit before 8-bit, with the float formats (BF16, FP16) at the end — so the sensible choice for a handset is the first one you see.

Every leaf shows a download size, and the catalog goes to some trouble to make that number honest. When the Hugging Face API returns the safetensors expansion — a map from datatype to parameter count — the size is computed exactly: parameters times bytes per element (4 for F32, 2 for F16/BF16, 1 for the 8-bit types, and so on).

When the API does not populate that field, the catalog falls back to an estimate calibrated against real mlx builds, and the calibration is more interesting than you would guess:

Tier	Bytes per parameter
2-bit	0.34
3-bit	0.45
4-bit	0.56
6-bit	0.80
8-bit	1.06
BF16 / FP16	2.1
FP32	4.1

Notice 4-bit is 0.56 bytes per parameter, not the naive 0.5 — real quantized checkpoints carry unquantized embeddings, per-block scale factors, and metadata. The constant is fitted so that, for example, a Qwen 1.5B at 4-bit estimates to about 837 MB, which is what the real download costs. A size label that is 12% short of the truth is the kind of small lie that erodes trust in every other number on the screen.

Alongside size, each curated model carries a minimum-RAM recommendation (minRAMGB, 6–8 GB for the shortlist below) used by the download prompt — a recommendation rather than a hard wall, because peak memory depends on context length and what else the phone is doing.

6. The device gate and the curated shortlist

All of the above decides what is offered. One check decides whether MLX download is offered at all: isDeviceCapable, true only on a real Apple-silicon device running iOS 27 or later. The simulator cannot execute MLX kernels, so there every MLX model reports a terminal unsupported state — the browser still renders, but nothing downloads.

The catalog also keeps a second, much smaller list beside the big tree: OnDeviceModelCatalog.curated, a hand-vetted shortlist of four 4-bit models for one-tap download — Qwen2.5 1.5B and Llama 3.2 3B for text, Qwen2-VL 2B and Gemma 3 4B for vision, spanning roughly 1.1 to 3.2 GB. The full hub tree is for exploring; the shortlist is for people who just want a good answer. Both funnel into the same download machinery.

7. Custom repos: the escape hatch

Curation is a default, not a cage. The browser accepts a hand-typed repo id, and the validation is deliberately minimal: isValidRepoID checks only that the string is a plausible owner/name — exactly two non-empty parts, id-safe characters. No network call confirms the repo exists; a bad id simply fails at download time with a clear error. Added repos are persisted (as a small JSON list in settings), listed beneath the curated catalog, and de-duplicated idempotently.

Two honest edges of this door. First, a custom repo bypasses the vision-and-tools gate — by typing an id yourself you are asserting you know what it is, and the fallback floor (section 10) catches you if you were wrong. Second, a custom model's modality is provisionally labeled text: the runtime auto-detects the real modality when the weights load, so the label is a placeholder, not a verdict. Removing a custom repo deletes its weights, and if it was the active model, the selection snaps back to the built-in Apple model — our first sighting of the floor.

8. The download lifecycle: `DownloadState`

Once a model is chosen, OnDeviceModelStore — the lifecycle layer between the catalog and the router — tracks it through a five-state machine:

enum DownloadState: Equatable {
    case unsupported            // not a capable device / OS for MLX
    case absent                 // capable, not yet downloaded
    case downloading(Double)    // 0…1 progress
    case ready                  // weights present (or the built-in system model)
    case failed(String)
}

 unsupported        (terminal on this device — the hardware gate said no)

 absent ──download()──▶ downloading(p) ──success──▶ ready
   ▲                       │       │                  │
   │◀────── cancel ────────┘       └── error ──▶ failed
   └──────────────────── delete() ────────────────────┘

States are seeded at app launch: on a capable device each MLX model starts ready or absent depending on disk; on an incapable one, unsupported. The built-in Apple model never enters the machine — asking for its state always returns ready.

A download runs inside a tracked task, keyed by model id, so it can be cancelled mid-flight. Under the hood the MLX runtime fetches the repo from the Hugging Face Hub with genuine 0-to-1 progress, then loads and caches the model container in memory — the same container later used for inference, so the download doubles as a warm-up. Two small concurrency guards in this path are worth naming, because each one is a bug that was lived rather than imagined:

A late progress tick must not stomp the finish. Progress callbacks hop across threads; a straggling progress(1.0) arriving after completion could overwrite ready with downloading(1.0) — a bar stuck at 100% forever. So progress updates apply only while the state is still downloading.
Cancel wins arguments. Cancelling sets the state to absent, and neither the completion path nor the error path is allowed to resurrect it to ready or demote it to failed. Cancel also removes partially written files, so the next attempt starts clean.

The subtlest choice is what "downloaded" means. The store does not scan a directory — it keeps a persisted list of ready model ids, marked only after a load has actually succeeded. The reason is ownership: the Hub client stores weights in its own cache (Caches/huggingface/hub/models--<org>--<name>, plus a .locks sibling), a location the app does not control, so a directory check against the app's own folder would see nothing. The flip side is that deleting a model must chase the weights to where they really live: delete() removes those Hub cache directories, evicts the in-memory container, and clears the marker — all three, or "re-download" would be a lie that completes instantly against stale files.

One deliberate ordering quirk: you may select a model before it finishes downloading. The browser shows it as your choice while the bar fills — and the agent, meanwhile, keeps answering. How? The floor.

9. A choice that follows you

A small but telling feature rides on top of this machinery: your model choice is synced across your devices. Picking a model calls a hook that fans the choice (model id plus a vision flag, which drives the chat composer's image-attach button) to your other signed-in devices over the account's encrypted sibling-sync channel. Each sibling adopts the selection without echoing it back — the apply path is marked non-propagating, the standard trick for breaking sync loops — and, if the model is an MLX download it does not have, kicks off the download on its own. Your phone and your iPad converge on the same brain without either one asking.

10. The permanent floor: the built-in Apple model

Everything above can fail. The network can drop mid-download. A custom repo id can be a typo. A device can be incapable. The user can delete the very model they had selected. The reason none of this is scary is a single model with special status: OnDeviceModelCatalog.appleSystem — the Apple Intelligence system model that ships with the OS. Zero download bytes, zero RAM requirement, no state machine: it is defined as present.

What makes it a floor rather than merely a default is that four independent layers all land on it:

Identifier resolution. Turning a stored model id back into a model tries the built-ins, then the custom repos — and returns the system model for anything unrecognized. A deleted custom repo's dangling id degrades safely instead of crashing settings.
Removal. Removing the custom repo you had selected resets the selection to the system model explicitly.
Session construction. AppleFoundationProvider — the provider that runs on-device inference — checks at session-build time whether the chosen MLX model is actually downloaded and loadable, and silently builds a system-model session when it is not. The comment in the code states the contract plainly: fall back to the system model so the agent always responds. This is what makes select-before-download safe.
Routing. LLMRouter — the single point that decides which provider every agent call site gets — constructs the on-device provider as the fallback arm of every route: no OpenAI key, no Anthropic key, a stale Homebase advert, a revoked shared runtime — each missing prerequisite resolves to on-device rather than to an error (the full ladder is Mechanism: External Routing & Fallback).

Why does an agent harness need this invariant so badly? Because an agent is a process, not a request. A chat app that fails one send shows a retry button. An agent that loses its model mid-plan strands a half-executed plan — tools called, state changed, no brain to continue. The router's promise that resolving a provider never fails is only honest because one model cannot be absent; every other arm of the bench is allowed to be conditional precisely because this one is not.

The floor even covers the vision path where the hardware allows: on an iPhone 17 or newer (and on iPads and Macs), the built-in model accepts images too, so falling back does not necessarily mean going blind.

11. What to take away

The browser's calm is manufactured, gate by gate. Three filters decide what you may see (loadable architecture; nameable by the parser; tool-calling and vision so the agent can drive it), two numbers keep the offer honest (exact-or-calibrated sizes, a RAM recommendation), one hardware check decides whether downloads exist at all, a five-state machine with two concurrency guards governs the lifecycle, and one undeletable model backstops every failure path.

The moral, in the spirit of this series: curation here is not taste — it is an availability contract. Every row in the browser is a model that will load, that the harness can drive, and that the device can hold; and if any promise breaks anyway, the fallback is a model that was never allowed to be missing.

References

Hugging Face Hub API — the /api/models endpoint, pipeline tags, and the safetensors metadata expansion.
The mlx-community organization — the source of every downloadable model in the browser.
Apple's MLX and the mlx-swift language-model runtime — the on-device execution layer whose factory registries define supportedModelTypes.
Apple Foundation Models — the framework providing the built-in system model and the session API both the system and MLX paths share.
Wikipedia: Quantization (machine learning) — background on precision tiers, for the curious; the harness only cares about the bytes.

Next: Mechanism: External Routing & Fallback.