← androux.org

How Mneme Remembers

The architecture of a self-hosted memory system for AI agents, explained as a workflow: one semantic substrate, typed arms for structure, and a human trust boundary that decides what memory is allowed to steer behavior.

Mneme · memory system 65 MCP tools · 30 REST endpoints PostgreSQL/pgvector · local embeddings 593 tests · ~50K thoughts

Mneme is the shared long-term memory behind every agent in my homelab: coding agents, a household voice assistant, shell scripts. They all read and write the same substrate. The core problem it solves is not storage, it is trust: an agent that "learns" things about you will eventually learn something wrong, and a memory system with no trust boundary turns one bad inference into permanent behavior.

Three ideas carry the whole design. Everything is a thought: every fact, event, and observation lands in one table with a vector embedding, searchable by meaning. Structure is a view, not a silo: typed "arms" (calendar, maintenance, household facts, training) store relational columns for exact queries while staying linked to their thought row for semantic recall. And memory is governed: what an agent learns is evidence until a human ratifies it. The system is a mirror of me that I curate, not a cage that trains itself around me.

Lane 1 · the write path: from noise to memory

triggers · many
Everything is a thought
Agent sessions, the voice assistant, brain-dumps, session transcripts, vault documents. Any source, one intake.
ai · router
Decompose & classify
An LLM router splits free text into clauses and types each one. "furnace filter Saturday, dentist Tuesday 2pm" becomes a maintenance task and a calendar event.
switch · by type
Route to a typed arm
calendar / maintenance / household facts / training / tasks, or a plain thought when nothing structural applies.
write · relational + semantic
Typed row + thought row
The arm stores real columns (due dates, intervals, quantities) for exact queries; a linked thought row keeps the same fact findable by meaning.
embed · local
Vector index
A locally hosted embedding model maps every thought into a 768-dim meaning space; pgvector + HNSW makes it fast. Nothing leaves the network to be remembered.
The typed-arm pattern

Structure and meaning are usually a trade-off: a calendar table can't answer "what did I say about the furnace?", a vector store can't answer "what's due next Tuesday?" Arms do both, because every typed row keeps a semantic twin.

Lane 2 · the read path: recall with a trust boundary

trigger · mcp / rest
An agent asks
recall("what does Andrew think about X?"). Any agent, any machine, same memory.
search · hybrid
Meaning + structure
The query is embedded and matched against the thought space; typed arms answer the exact parts (dates, counts, intervals).
if · governance gate
Ratified by a human?
Every governed memory carries a trust tier. The gate is the load-bearing part of the whole system.
confirmed ↓
may act
Steers behavior
Human-ratified. Agents may treat it as true and act on it.
evidence ↓
informative only
Context, not command
An agent's unratified observation. Visible, useful, never steering.
audit
Every recall is traced
Who asked, what surfaced, which tier it came from. Memory you can't audit is memory you can't trust.
Mirror, not cage

Agents learn things about you constantly, and some of it is wrong. In Mneme a learned fact is evidence until I ratify it in review. The self-model stays a mirror I curate, never a feedback loop that quietly decides who I am.

Lane 3 · background loops: the system that keeps learning

queue · async
Derivation queue
Slow jobs (transcript mining, re-syncs) run through an idempotent work queue instead of blocking any agent's request.
mine · transcripts + backlog
Judgment corpus
Hundreds of real decisions, kept with the options that were rejected. New agents can ask "how does Andrew usually rule on this?" instead of guessing.
gate · human review
Surprise & ratification
When I override a recommendation, the system queues a "why?". Pending memories wait in a review queue for confirm / reject / edit.
surface · everywhere
One memory, many mouths
65 MCP tools for agents, 30 REST endpoints for scripts and the voice assistant, daily briefs for me. Same substrate under all of it.
Why it's built this way

Code can be rebuilt cheaply; understanding can't. The thing worth persisting across agents, sessions, and machines is the judgment and context, so that is what the architecture optimizes for. Everything else is replaceable plumbing.

Companions