Your phone holds the most complete record of your relationships that has ever existed: every message, every call, every plan, every "we should really catch up." Pixie ships a personal agent that reads that record and helps you actually keep the people in it close. This is the story of how you build that agent so the record never has to leave your hands.
This article is the entry point to a series, and the series is the third pillar of the system.
Two of the other three are about moving bytes privately: Contact Discovery is how friends find each other without the server learning who knows whom, and The Relay is how messages travel without the server learning who is talking. The fourth, GroupThink, is how the devices on your account behave as one mind: each holding a full replica, none of them the original.
This series is about the cognition that rides on top of them: the agent. Each hard sub-problem gets its own deep dive, and this article links out to them as we go.
One scoping note before we start. This series is about the agent harness: how models are chosen, bound, decomposed, extended, and shared. It is not about machine learning.
We will treat the language model the way you treat a database in a systems course: a component with an API, a latency, a context budget (how much text it can be shown at once), and failure modes. If you are comfortable with programs, APIs, and a bit of systems thinking, you have everything you need; you never need to know what happens inside the model.
Pixie is a relationship-care app. Its one job is helping you stay close to people: the friend whose exam was yesterday, the group trip that never gets scheduled, the parent you keep meaning to call. Social software has historically attacked this job with feeds ranked for engagement, and the result is well documented: more scrolling, not more closeness.
Pixie's bet is different: put a personal agent on your side of the glass. It reads what is already there (your conversations, your calls, your posts, your calendar) and surfaces the small, timely thing worth acting on: call Priya; text the gym group back about Saturday; it's Rohan's birthday and you haven't talked in a month.
What do we mean by "agent"? Strip the buzzword and it is a simple loop:
while task not done:
model reads context (the conversation so far + tool results)
model either answers, or asks to call a tool
harness runs the tool, appends the result, loops
A language model that can request actions (search my memory, read that thread, check the calendar) and a harness that executes those requests and feeds results back. The model supplies judgment; the harness supplies memory, tools, permissions, and the rules of the game. Everything interesting in this series lives in the harness.
Two facts make this particular agent hard to build honestly:
SpeechTranscriber; audio and transcripts
never leave it.)Almost every AI assistant on the market resolves this tension in the same way: upload the data, run the model in the cloud, promise to behave. We resolved it the other way.
The naive architecture is a thin client and a smart server: your phone does little beyond streaming context up to a hosted model, and the platform's servers do the thinking. It is the easy road: big models, no on-device constraints, one place to iterate. But it also means the platform holds a live feed of your relationships. And your "personal" agent is a rented one: cancel the subscription and your assistant, with its memory of you, evaporates.
We know the pull of that road because we walked partway down it. Phase 2 shipped a
bundled-inference tier (llm-svc, a thin proxy that forwarded a device's
language-model calls under the platform's account). We eliminated it. Not deprioritized: eliminated. No
tier of the product routes cognition through our servers, and there is no agent runtime
on the server side anywhere in the system. The rule that replaced it is the trust
decision this whole series unpacks:
Local-first cognition: the agent comes to the data, not the reverse. Your memory, your relationship map, and the reasoning over them live on hardware you own. What leaves the device leaves sealed.
Saying "local-first" is cheap; the architecture only becomes trustworthy when the expensive consequences are accepted. There are three, and they shaped everything:
We want to be honest about the price. A hosted frontier model (one of the biggest commercial models, reachable only through a paid service) is smarter than anything that fits in a phone. A large part of this series is the engineering spent closing that gap: decomposition, routing, and an escape hatch that lets you borrow big-model power from hardware you own (your Mac) without breaking the rule.
The bet is that for a personal agent, custody beats raw IQ: a modest intelligence that knows your whole context and answers to no one else is worth more than a brilliant one on someone else's computer.
The whole system is a clean horizontal cut, and the cut is the product thesis. On top sits the personal cognitive layer: memory log, grep retrieval, planner, tools, skills, the Notebook. That layer is this series: it lives on your devices and is yours permanently. Under it, the social relay layer (The Relay and Contact Discovery) is sealed-sender transport: the server routes envelopes it cannot read, from senders it cannot identify.
At the bottom, the model execution layer is the swappable model bench: Apple's system model, local MLX models, your API keys, your Mac. Never platform-owned.
The design invariant is that layers above can swap layers below without losing identity. Change the model (Layer 3) and your agent is still your agent: same memory, same relationships, same behavior contract. That invariant is why the no-embeddings rule exists, and it is what makes "the platform never owns the cognition substrate" a structural claim instead of a promise.
Layer 1 is the agent proper, and it is what The L1–L2–L3 Harness, Skills, Tools, and the @-Namespace, and The Agent in Your Day dissect: the append-only memory log, the read-only tool registry, the planner, the user-extensible skills, and the Notebook (the agent's distilled, mostly user-legible memory of you and your people).
Layer 2 is the transport the agent rides, and the Relay and Contact Discovery series own it. The one term you need from them here is sealed sender: every envelope on the wire hides its sender even from the relay, and the recipient is hidden too. A 1:1 message is not addressed to anyone's public key. Instead it rides an opaque rotating pair subject: a label both friends derive from the channel key they already share. The subject rotates every hour, and it is bound to the specific recipient device, so each of a peer's devices has its own subject. No phone number or account id enters the derivation, and the key material is the shared secret, so the value is opaque to anyone who does not already hold that secret. The relay sees that opaque subject and a little coarse delivery metadata: never who is talking, and never to whom.
Two properties of that layer matter constantly in this series. First, anything the
agent sends anywhere (including to your own other devices) goes as a sealed envelope,
so building a new agent feature never means giving the server a new window. Second, your
devices form a fleet: an iPhone, an iPad, a Mac, all holding the same memory. The
fleet is kept coherent by a sealed same-account sibling channel called sync_delta:
the wire kind that carries notebook and memory deltas, state
digests, and, as we will see, even
inference traffic between your own devices. A seat-token election called the driver's
seat (one of GroupThink's
mechanisms) ensures exactly one device runs autonomous agent work at a time, so your
agent never shows up twice to the same conversation.
Layer 3 is the model bench, and it is deliberately a bench, not a model: a set of
interchangeable providers behind one Swift protocol (an interface contract) named
LLMProvider, with one routing
authority deciding which arm thinks for any given task. That is Choosing
Brains.
Everything below Layer 1's surface falls out of five questions. Each is an article; the deeper "Mechanism" pieces under them are the engine room.
| Question | Deep dive |
|---|---|
| Which model does the thinking, and who controls that choice? | Choosing Brains: the Model Bench |
| How does a small on-device model do reliable multi-step work? | The L1–L2–L3 Harness |
| How do you extend and steer the agent without retraining anything? | Skills, Tools, and the @-Namespace |
| How do you borrow big-model power without renting a cloud? | The Homebase |
| Where does all of this actually meet your day? | The Agent in Your Day |
A quick tour, so the map has terrain.
Choosing brains. Every model call in the app funnels through LLMRouter, the one
component allowed to decide which provider thinks. The bench it routes across: Apple's
built-in system model (the floor, always present on a supported device); a curated
catalog of open-weights models (ones anyone can download and run) from the
mlx-community hub, running locally on MLX (Apple's framework for executing models on
Apple-silicon hardware); your own OpenAI or Claude API keys (personal credentials that
let the app call those services on your account) as an explicit opt-in; and
external models served from your own Mac. The user-facing truth is a single
model-source radio in Settings: three positions, exactly one selected, like the
station buttons on an old car radio. The radio is the only
routing authority, so picking a model in some sub-pane can never silently reroute your
cognition. A missing prerequisite degrades to on-device rather than blocking: no key at all,
no verified runtime, or a Mac that has stopped advertising all resolve on-device before the
call is even made. (A key that is present but rejected is a different case: that surfaces
as an error on the turn, because the router cannot know a stored credential has gone stale
until the provider says so.)
How the catalog decides which community models qualify (tool-calling and vision gates,
quantization tiers, device capability) is Mechanism: On-Device
Curation; the external arms and their fallback
ladder are Mechanism: External Routing & Fallback.
The L1–L2–L3 split. A tool's schema is the machine-readable list of arguments a
model has to be shown before it can call that tool correctly. Ask one small model to
plan a task, follow the plan, and wield a toolbox of full JSON schemas all at once, and
it drops threads. We measured it, and the monolithic executor lost badly. So the runtime
splits the cognition three ways. An L1 planner writes a short free-text plan. An
L2 skill-follower executes the plan while
seeing only one-line signatures of the tools (not their schemas) plus a single
meta-tool. L3 tool-agents are fresh, disposable sessions, each bound to exactly one
real tool's full schema for a single call. No context ever carries more than one schema.
That is what keeps the harness reliable today, and it is the scaling hedge for a
many-tool future. The single meta-tool trick (CallTool, the one door through which every L3
dispatch passes) is Mechanism: CallTool, One Tool to Bind
Them.
Skills and the @-namespace. A skill is a user-ownable bundle of instructions
("how I like my weekly review done") that the planner can pick up by name. Skills and
tools live in one shared @-namespace (@search-info, @check-schedule, …). Sharing
one namespace means a skill can reference tools and other skills by handle, and renames
cascade instead of dangling. The planner's menu of @-entries also doubles as the user's
alignment surface: what the agent can reach for is exactly what you can see listed.
Skills travel, too: you can send one to a friend as a sealed message, closure and all
(the closure being the skill plus everything it references, and everything those
reference), to be reviewed before
import. That path (the transitive @-closure, collision re-slugging, review-before-import)
is Mechanism: Skill Bundles & Sharing.
The Homebase. The rule was never "small models only." It was "hardware you own."
Your Mac runs the same app as a full peer (same keys, same memory, same agent) and
becomes the Pixie Homebase. It hosts big open models via Ollama (a local runner that
downloads model weights and serves them behind a plain HTTP API) and serves your handset
over the sealed relay: an infer_request / accept / chunk / done round
riding sync_delta, with timeout watchdogs at every stage and a silent on-device
fallback wired inside the provider. A sleeping Mac is a quality downgrade, not an
error (Mechanism: Inference over the Relay). The Homebase can
lend that power to people you trust under revocable grants whose usage metrics count
bytes and tokens but structurally cannot contain content (Mechanism: Sharing a
Homebase). And it opens the one governed door for
other agents to drive Pixie: Claude Desktop, a local model, anything speaking MCP (the Model
Context Protocol, the emerging standard for exposing tools to agents).
That door is a loopback-only server, reachable only from programs on the same Mac. Its
read tools project only the unsealed zones, and its three
write tools suspend on a per-call human approval sheet showing the exact draft, where
dismissing means deny (Mechanism: MCP, Letting Other Agents
In).
The agent in your day. Where it all cashes out. The chat tab. Summon is the @pixie inline
tag: mention the agent inside a human conversation and it answers only on your
screen; the peer never sees the summon. (A single chokepoint refuses summon-shaped
bodies at every peer-send path, and property tests, which fire a barrage of
generated inputs at a rule, hold it there.) Proactive nudges arrive in the right
conversation at the right moment: "how was your day?" fires when you actually get home,
not at 6 PM sharp. Huddles are where the agents of a mutually-trusting friend group
talk to each other in structured rounds over group E2EE and come back with at most a
nudge; you can watch the exchange while it runs, but nothing of it is kept afterwards. Plans turn a huddle
into a calendar entry. And the Notebook is the agent's curated memory: a clear zone
you can read and edit (your edits are ground truth) and a sealed zone for what the
agent learned in confidence from other people's agents, encrypted under a key you don't
hold, with the leak direction structurally forbidden. That surface (and the memory
machinery under it) is The Agent in Your Day and
Mechanism: Memory & the Notebook.
To make the layers concrete, follow a real question through the harness. You ask the
chat: "What did I promise the gym group last month?" The LLMRouter reads the
model-source radio and binds a provider (say, an mlx-community model running locally).
The L1 planner writes a free-text plan: check the notebook and search memory for
gym-group commitments, then summarize. The L2 skill-follower holds that plan, one-line
tool signatures, and a single meta-tool, and turns the plan into two call_tool
dispatches. Each lands in a fresh L3 session bound to one full schema: grep_memory
regex-greps the on-disk text mirror of the event log, check_notebook searches the
Notebook's clear zone. The answer is assembled from what they returned, with sources.
Count the network calls along the way: zero. The model is local, the memory is local, the tools read local stores. If the radio had pointed at your Homebase, the prompt would have crossed to your Mac as a sealed envelope between your own devices: still nothing a server could read.
The shape of this trace (why L2 never sees a schema, why L3 is disposable, why the answer is assembled from tool results rather than the model's paraphrase of them) is the substance of The L1–L2–L3 Harness.
A harness is defined as much by its refusals as its abilities, and Pixie's sharpest design choices are refusals:
grep_memory, read_conversation,
read_post_thread, read_call, check_notebook, file and calendar lookups: the
agent reads and plans. It never sends a message, never posts, never mutates your
relationship data. One-tap call/text actions are plain app UI. The Feed itself is an
ordinary reverse-chronological view of the posts your contacts sent you: no engagement
ranker sits between you and them. The agent's own cards are generated rather than
ranked. And every tool the agent invokes shows up as a visible pill in the thread, so
"what did it look at" is answerable from the transcript.Notice the pattern: every boundary is structural (a missing tool, a missing key, a chokepoint in code) rather than an instruction the model is trusted to follow. Prompts are how we ask the agent to behave; architecture is how we guarantee it.
Read top-down for the story, bottom-up for the mechanisms:
A recurring theme, and really the moral of the pillar: the hard part of a personal agent is not intelligence. It is custody. Models improve on their own schedule, and a swappable bench inherits every improvement the day it ships. What no model release can retrofit is where the thinking happens and who owns the memory it thinks with. Get the harness right and the agent gets smarter every year without your life going anywhere at all.