🧠 Five Attempts at Giving an Assistant a Memory

In my last post I wrote about the workflow tooling I use to direct coding agents. This one is about the other side of the same interest, which I have been failing at for considerably longer: building myself a personal assistant that actually remembers me.

I have made five attempts at it now. The assistant itself is not really the point, because everyone has built one of those by now. What I want to write about is that every version ran into the same wall in a different way, and it took me until the fifth to understand that the wall is memory and nothing else.

The Origins

The goal has always been Jarvis, and I do not mean that as a joke. The thing that stuck with me from those films was never the suit, it was that he talks to his house and his house knows things.

I have been chasing that for a long time, in whatever the technology of the moment allowed. Long before any of this was possible I had regex-driven automations wired up to Google’s speech to text, where you said a phrase, a pattern matched, and a script ran. There was no understanding involved at all, just string matching with a microphone in front of it.

Later there was Agentic Agency, which was meant to be an automated software agency and synced Slack, GitHub, GitLab, Jira and Linear through background workers into one place. The assistant part of it barely existed. The syncing part is the half I kept, and it taught me the thing that turned out to matter most, which is that an assistant is only ever as good as the data it can see. Getting data in, and keeping it in sync in both directions, is a much bigger problem than the model on top of it.

The Five Attempts

• A prototype, September 2024, which lasted a single day. It got far enough for me to learn that “the assistant remembers things” is not one feature.
• MoaiMind, over about five weeks around the end of 2025. A platform with the problem split into named pieces: a capture engine, a knowledge engine, a decision engine, an execution engine.
• Donna v1, over nine days in March 2026. Everything is a file.
• Donna v2, from March to the end of July 2026. A rewrite, started the same day I stopped working on v1.
• The current one, which lives in my main monorepo alongside everything else I run.

With MoaiMind I split the problem into named engines and thought that was progress. It was not, because calling something a knowledge engine told me nothing whatsoever about what was supposed to go inside it. All of the difficulty lived in that one box and the name concealed it.

The Vault

Donna v1 is the version whose storage instinct I still keep, and it is the one I enjoyed building most.

There was no database. Everything it knew was a plain file, JSON and JSONL and Markdown on disk, with a SQLite full-text index on top. Somewhere in the middle of that week I migrated the JSON manifests over to Markdown with YAML front matter, which is the point where it stopped being a data directory and properly became a vault.

The reasons for that hold up well:

• I can read my assistant’s memory in a text editor
• I can grep it
• I can diff it and see what it decided to remember yesterday
• Backup is cp -r
• If the project dies, the data does not, because the data was never in a format that needed the project

Around it went the parts that actually make one of these usable day to day: channels for web, Telegram, Slack, Discord and WhatsApp, integrations for Google Calendar, Gmail, Drive, Outlook, OneDrive and GitHub, cron and event triggers, secrets encrypted at rest, and danger scoring, where every action the assistant wanted to take was scored for risk and checked against approval rules before it ran rather than after.

Then I used it for a month and ran into something full-text search cannot fix.

The Memory Problem

When I asked it something it retrieved matching text, and my memory is not a pile of matching text.

Some of what it had stored used to be true and was not any more, and retrieval returned both versions ranked by similarity with no idea which one was current. Some of it I had said once, in passing, and it carried the same weight as something I had repeated for a year. And when it was confidently wrong about something, I had no way at all to ask it where that had come from.

Donna v2 answered that by storing claims rather than text. Each claim is attached to the evidence it came from, with provenance, and a claim is not overwritten when a newer one contradicts it. Both exist, both are dated, and the newer one supersedes rather than deletes. So it can tell me that I said one thing in March and a different thing in July, instead of confidently telling me the second one and quietly losing the first.

With claims and evidence I can ask where something came from and get an actual link back. With full-text search I could only ever get another paragraph.

The Landscape

Before starting the fifth version I went and looked properly at what everyone else had done, because writing your own memory layer is the kind of decision that should have to justify itself.

• QMD - Markdown-first local search, BM25 with vectors and reranking, usable as a CLI, an MCP server or a library
• Basic Memory - a local Markdown knowledge graph that the model reads and writes bidirectionally
• MemPalace - verbatim local storage with scoped retrieval, deliberately not summarising
• Graphiti - temporal facts with provenance episodes and hybrid graph retrieval
• Mem0 and OpenMemory - memory types, scoping, access logs, multi-signal retrieval
• cognee - ontology, permissions, a managed world model
• SuperMemory - retrieval and extraction behaviour worth borrowing
• Plus HydraDB, OpenViking, Engram, Memgraph, Kuzu, LanceDB and Qdrant underneath

There is genuinely good work in that list, and several of them are solving a harder version of the problem than the one I have.

The Decision

I did not pick one. What I wrote in my own architecture notes was to not choose a dedicated memory backend at all yet, and instead define a memory contract and let the runtime attach providers per workspace.

The path is local files with ordinary search first, then a provider interface over those files, then optionally a QMD-shaped retrieval adapter, a SQLite full-text cache, a file graph, vectors. Managed memory infrastructure stays an option rather than becoming the default.

The reasoning is that everything interesting in that list above is semantics rather than storage. Structured memory types, graph-aware recall, recency, access logs, usefulness ranking, all of it can be implemented over local files and delegated to a provider later if it earns its place. Adopting a hosted memory backend on day one buys those ideas at the cost of no longer being able to read my own memory without somebody else’s software. Embeddings are optional, and a vector database should not be required for an assistant to remember that my sister’s birthday moved.

The Current Version

The fifth one lives in my monorepo and behaves less like a chatbot than like a local chief of staff. It opens on what matters today rather than a blank prompt, and the loop it runs is deliberately visible: capture, source reference, extract, review, write, retrieve, explain, follow up.

The rules it is built on are the ones the first four earned:

• Files remain the source of truth. Every durable item can be inspected outside the app.
• Memory is reviewable. Inferred facts, entities and patterns arrive as candidates rather than as knowledge, and I promote them.
• Context is visible. An answer shows the sources that shaped it, or says explicitly that it had none.
• Automation is inspectable. Runs, triggers, writes, errors and approvals all have history.
• Scopes are explicit. Personal, company, client and project context do not merge silently.

That third rule is the whole journey compressed into a line. The first version could tell me things, the second could tell me why it thought them, and this one has to admit when it has no source at all, which has turned out to be the more useful behaviour of the two.

The Key Learnings

• “Remember” is not one feature. All five versions could store things. Only the later ones could be interrogated about what they had stored, and that was always the actual requirement.
• Danger scoring was worth it from day one. Scoring actions for risk before executing them never once got in my way.
• Plain files are underrated. The one decision I have never revisited across five rewrites is that my own data should be readable without the software that wrote it.
• Inferred memory belongs in a review queue. An assistant that quietly writes its own guesses into its own memory will eventually be confidently wrong about my life, and I will have no idea when it started.
• Building the wrong one first was not waste. I did not reason my way to any of this. I built a search index, lived with it, and found the questions it could not answer.

None of the Donna repositories are public, and the current version is part of a private monorepo, so there is nothing to link here yet. MoaiTime, which was an ancestor of all of it, is open source.

The End

Most people building one of these pick a vector store and move on, which is exactly what I did the first three times. It gets you retrieval, and I kept assuming that was the same thing as memory.

What I did not expect going in was how much of the answer would turn out to be about review rather than storage. I thought I was solving a retrieval problem, and I think now it is closer to a bookkeeping one.

More of what I have built is on the projects page, and if you want to talk about agents that have to be right rather than impressive, here is what I am looking for.

I hope you enjoyed this blog post and I will see you in the next one!