An agent that helps you care for your relationships is only as good as what it remembers. And an on-device model has a context window (the fixed amount of text a model can consider at once) of a few thousand tokens, where a token is roughly a word-sized chunk of text. Set that against a life that produces years of messages, calls, posts, and journal entries. This article is about the machinery that resolves that mismatch: an append-only log the agent can grep (search with text patterns, like the classic Unix tool), a distilled Notebook split into a zone you can read and a zone you never can, a reaction that lets you pin a moment straight into memory, and a retrieval path that is a tool call, never a context dump.
This is the deepest mechanism dive under The Agent in Your Day, which tours every surface the agent shows up on and leans on this one constantly: the chat that knows your friend's news, the brief that knows what to nudge about, the huddle that knows what to bring.
Here we open the hood on the memory itself: how it is written, how it is protected, how it is searched, and how it can even live partly on another one of your devices without the agent losing track of it.
Start with the naive design, because it is what almost every chat-with-memory demo does: keep the agent's notes in a big string and prepend it to every conversation turn. "Context stuffing." For a cloud model with a two-hundred-thousand-token window, you can get surprisingly far this way. Pixie cannot, for two reasons: one of budget and one of principle.
The budget reason: Pixie's default brain is Apple's built-in system model, with a context window of roughly 4k tokens (see Choosing Brains: the Model Bench). Every token the Notebook occupies is a token the actual conversation, the tool results, and the reply cannot use.
Stuff a few dozen notes into the system prompt and the model has no room left to think, and most of those notes are irrelevant to the question at hand anyway. You asked about Priya; the model is carrying your dentist's phone anxiety in its working memory for no reason.
The principle reason: what rides in context is invisible and unaccountable. A tool call is an event: it can be logged, surfaced in the UI ("Checked your notebook"), rate-limited, and reasoned about. When memory access is a tool call, "what did the agent look at to produce this answer?" has an answer.
So Pixie's rule is: memory is never injected; it is fetched. The system prompt
(the standing text the model sees before every conversation) carries the agent's persona
and instructions, and nothing of the Notebook.
When the model needs background, it calls a tool (check_notebook for the
distilled notes, grep_memory for the raw log), and only the handful of matching
entries enter the context, for that turn only. The rest of this article is the
machinery on the other side of those two calls.
Underneath everything sits the memory log: a stream of MemoryEvent rows in the app's local
database, one per thing that happened: a journal entry, a message, a call, a voicemail, a post,
a calendar event. It is append-only: events are added, never edited in place. That property is
not incidental.
An append-only log is trivially safe to sync between your own devices: two devices merge by "insert-if-absent", so there is no conflict to resolve. And it is an honest historical record: what the agent believed last March is still there, not overwritten by what it believes now.
But a database of rows is not something a language model can browse. The searchable
form is the MemoryTextMirror: a derived, plain-text index built from the
log, existing purely so that a regular-expression search (a text-pattern query) over it is fast. Derived
means disposable: if the mirror is empty (say, after a cold launch), it is rebuilt
from the log on first use, so a search never silently misses.
The agent's handle on all this is GrepMemoryTool, a native tool the model can
call with a keyword, name, or phrase. Its name is the tell: this really is grep,
pattern matching over text, with no embeddings (numeric vectors
that capture a text's meaning), no vector database, no semantic index.
The model's query is tried as a regular expression; if it is not a valid pattern
(models love stray parentheses), it falls back to a literal search. Three details
in its implementation are worth noticing, because each one is a scar from a real
failure mode:
Positionally, grep_memory is the last resort. It is broad and old (the place
to find something you mentioned in passing months ago), and it is one of the
sources bundled into the Search Info skill (see
Skills, Tools, and the @-Namespace). For most
questions the agent should not need to grep raw history at all, because there is a
distilled layer above it.
The Notebook is what the agent actually reasons from: a small set of curated notes distilled out of the memory log, surfaced in the app under You → Agent → Notebook. Where the log is the firehose, the Notebook is the digest: roughly one page per person in your life, plus a handful of standing notes. Its note kinds, as the code defines them:
The defining structure, though, is not the kinds. It is the two zones.
The clear zone holds everything above: readable, editable, yours. The sealed zone is the agent's private workspace: the prep notes it takes into a huddle (the agent-to-agent sessions described in The Agent in Your Day), the takeaways it brings back, and anything it deliberately keeps to itself. The hard floor: anything learned from other people's agents in a huddle is unconditionally sealed, because it is second-hand context about your friends' lives, shared agent-to-agent in confidence. Rendering it in your Notebook would leak their private lives through your screen.
Sealed notes are encrypted with NotebookSeal: AES-GCM
(authenticated encryption, so a tampered or wrongly-keyed blob fails to open rather than
decrypting to garbage). The key is 256 bits, minted on first use, and held in the device
Keychain, Apple's encrypted credential store. That key is never surfaced in Settings and,
unlike the backup key, never exported.
Only ciphertext (the encrypted bytes, unreadable without the key) ever touches the database, so even iCloud sync carries bytes nobody but this agent can open. Wipe the app and the key dies with the Keychain: the sealed zone becomes permanently unreadable, by design.
But encryption is not the interesting control. The interesting control is
structural, and it lives in NotebookStore, the single object through which
every Notebook read and write flows. It exposes two disjoint read APIs: two entry points with no overlap. clearNotes and searchClear see zone == .clear
only, and feed the UI and the chat agent's check_notebook tool.
sealedNotesForHuddleReasoning decrypts the sealed zone, and is called only by the
non-user-facing huddle/nudge pass: the background reasoning that preps huddles and drafts
nudges, the gentle suggestions the agent surfaces on its own.
The user-facing chat agent's only path into the Notebook is check_notebook, which calls
searchClear, and searchClear filters to the clear zone before it does anything else. The
sealed zone is not "hidden" from the chat; it is structurally absent.
There is no reclassifyToClear method, and no method anywhere that returns a sealed note's
plaintext to a UI or chat caller. The boundary is enforced by omission, and pinned by a test
asserting that no clear-zone read can ever surface a sealed note (a sealed-note leak into the
visible Notebook is classed as a P0 trust breach, the highest severity).
Why structure instead of a prompt rule ("never reveal your private notes")? Because a prompt rule is a request to a language model, and requests to language models can be talked around. Structural absence cannot: the chat agent has literally nothing sealed in reach, so there is nothing a clever jailbreak could extract.
It even closes the metadata leak: learning something from the shape of an answer rather than its content. The agent cannot distinguish "no sealed note about Jimmy exists" from "one exists but is hidden," so a no answer reveals nothing either way. The huddle/nudge reasoning pass, which does read sealed notes, is not something you converse with; it emits finished nudges, never raw context.
Clear-zone notes are written by the notetaking distill loop (NotetakingService).
Periodically, on the agent's opportunistic foreground pass, it finds conversation threads that have accrued new material since a stored watermark: a single stored timestamp marking when the loop last ran. Per-thread incrementality comes from the idempotent upsert (an update-or-insert write that is safe to repeat) keyed on the thread id, not from the watermark itself.
It feeds each thread (capped at the most recent forty messages, for the same context-budget
reason as everything else) to whatever model the LLMRouter seam resolves; that is
the routing layer that decides which model handles a job (see
Mechanism: External Routing & Fallback). It asks for a
small structured result: a conversation summary, an optional summary of the person behind the
thread, any durable facts gleaned about you, and optional topic tags.
Each result lands through NotebookStore.upsertClear, and the upsert is where
the care went. Every note kind has an identity key: a Person page is keyed by its
contact, a digest by its source conversation/post/call ID, and "Notes about you" is a
singleton (only one such note ever exists). So re-running the loop updates in place rather than duplicating.
Distillation is idempotent: run it twice, get the same Notebook.
Two guard clauses in that upsert carry real weight:
userEdited, and the
distill loop will never overwrite it again. The agent's summary is a draft; your
version is ground truth.updatedAt timestamp is not touched. This sounds like fussiness, but the
timestamp is hashed into the
state digest (a short
fingerprint of all local state) that your devices
compare to stay in sync (the sync_delta machinery that keeps a phone and a Mac
converged). A loop that re-stamped an unchanged note on every pass would
make the devices disagree forever about state that is actually identical,
re-syncing in a storm. Convergence demanded that "nothing changed" leave no trace.Everything so far is the agent deciding what matters. Spark (the Pixie Reaction) is the inverse: you deciding. Long-press any message, post, or comment and, alongside the ordinary emoji reactions, there is Pixie's own mark. Applying it pins that moment into the Notebook.
Mechanically (NotebookStore.pinSpark), a pin creates a sparked note in the clear zone
carrying the pinned text, a reference back to its source conversation or post, the contact it
involves, and today's date.
Unlike distilled notes, every pin creates a new row: pins accumulate rather than upserting,
because collapsing all your pins from one thread into a single self-overwriting note would
defeat the point of pinning. Each pin also files a MemoryEvent in the log ("something the
user chose to remember"), so the grep path sees it too, without a Notebook lookup.
The property worth dwelling on: the spark is private by construction. An emoji reaction is a social act: it is sent over the wire and rendered on your friend's screen. The Pixie Reaction is a curation act. Nothing in its code path touches the compose or outbox machinery. No envelope (the encrypted packet a message travels in) leaves the device, and the person whose message you pinned never learns you pinned it.
The only network the pin ever rides is the sealed same-account sync channel to your own other devices, so your Notebook stays whole across your phone and your Mac. It is the difference between telling someone "that mattered to me" and underlining a sentence in your own diary.
A keyword search is a blunt instrument once the Notebook grows. "What did I learn about Ada's travel plans last month?" is not a keyword: it is a person, a topic, and a time window. So notes carry three structured facets alongside their text:
contactID: which person the note is about;topicTags: free-form topics ("travel", "health"), emitted by the distill
model or attached to pins;aboutDate: the date the note is about (the pin date, the call date),
falling back to when it was last updated.searchClear applies these as filters before keyword scoring: narrow to one
person's notes, one tag, one date window, then rank what is left. Ranking is
deliberately simple: a hand-rolled score, not a machine-learned one. A term
matching the title counts 3, a tag 2, the body 1. Ties break newest-first, and the top
eight come back. With an empty query the filters alone answer "everything about X":
precise recall with no free-text guessing.
The model-facing wrapper is CheckNotebookTool, a native tool whose arguments mirror those
facets: a query, an optional person (a name, matched case-insensitively against your
contacts and resolved to a contactID, with an honest "no notes filed under that person" if
the name doesn't resolve), an optional topic, and an optional daysBack that becomes a date
window.
Results come back labelled by kind (Person · Ada, Conversation · …, Sparked by the user (remember this)), so the model knows whether it is reading a relationship page or something you
explicitly pinned.
Two runtime guards wrap every call, and they exist because small models loop. A per-turn dedup (duplicate-detection) check refuses an identical repeat call ("You already checked the Notebook for that…"), and a per-tool call budget hard-stops the turn when a model keeps re-querying with cosmetic variations: a failure mode observed live, and one that prompting alone demonstrably did not fix.
When the tool runs inside an L3 tool-agent (the throwaway session spun up to craft a single tool call), those guards switch off locally because the L2 loop above owns them: the one-tool-per-session architecture explained in Mechanism: CallTool, One Tool to Bind Them. Each call also fires a UI hook, which is why the chat shows a small "Checked your notebook" pill. Retrieval-as-tool-call makes memory access visible, exactly as §1 promised.
One more trick, and it is the newest: a Notebook note's body does not have to live on your phone at all.
If you run a Homebase (the Mac peer described in The Homebase), a clear-zone note can be offloaded to it. No body is shipped: ordinary sibling sync (the always-on replication between your own devices) has already delivered the Mac its own copy. So the phone simply asks the Homebase to confirm it holds this exact body, by note id and content hash (a short fingerprint computed from the bytes, which changes if the content changes at all).
Only a verified acknowledgement lets the phone clear its own copy, and a Homebase whose replica (its local copy of your data) has not yet converged (caught up to the same state) refuses. That is why the rule is "never clear before ack" rather than "ship, then clear".
The local row then collapses to a pointer: the body is cleared, and in its place sit the
holder's identity and a content hash. Everything else (the title, the kind, the tags, the
aboutDate, the contact link) stays resident.
That remainder is exactly the retrieval index of §6, which means an offloaded note still
matches: check_notebook still finds it by person, tag, date, or title, and the Notebook UI
still lists it (marked "Offloaded — opens from your Homebase"). The index lives on the phone;
the bytes live on the Mac.
Opening one fetches the body back on demand: a sealed request to the holder, the
body streamed back, and a fail-closed check. Fail-closed means any doubt ends in
refusal rather than a guess: the returned bytes must re-hash to the
contentHash stamped at offload time, so a buggy or compromised holder cannot
substitute different content. Viewing is read-only and transient. Editing
requires explicitly bringing the note back to the phone first, because an edit
against a body this device doesn't hold would silently desync the parked copy.
The gates around this are small and strict, in NoteOffloadService:
updatedAt: the same timestamp discipline as §4. Where a note's body
lives is per-device state, not an edit; if it bumped the clock, every offload
would look like a content change and set the
cross-device reconciliation machinery
chasing a difference that isn't one.The holder enforces its own mirror-image gate: it serves a body only when the request names a clear note it actually holds and the resident body re-hashes to the requested hash, so neither side ever trusts the other's claim about content.
Step back and the memory system is four layers, each earning its place:
MemoryEvents, the honest record, merged trivially
across devices, searched by literal grep as the last resort.The moral mirrors the one that closes this series' cryptography siblings: the hard part of agent memory is not storing text. It is access discipline: making sure the right slice of memory, and only that slice, reaches a small model at the right moment; making the private parts unreachable rather than merely discouraged; and making every act of remembering something you can see, edit, and own.
CheckNotebookTool and GrepMemoryTool implement.One mechanism remains: Mechanism: Forming a Cohort Without a Server. For the full map of the harness this article belongs to, return to Agent.