Memory Architectures

Why this matters

The first system I built with long-lived AI memory failed in a familiar way: it could retrieve something plausible, but I could not explain why that record existed, which version of the source produced it, or whether a restart would preserve the same result. The model appeared to remember. The system did not have durable state.

I now treat memory as a set of data products with different contracts, not as a conversation transcript plus a vector index. A useful architecture separates at least four concerns:

• Working memory holds the bounded context assembled for one request.
• Episodic memory records immutable events: observations, tool results, user decisions, and system actions.
• Semantic memory stores normalized claims or entities that can be reused across episodes.
• Derived retrieval state contains chunks, embeddings, lexical indexes, summaries, and ranking features.

That separation supports the properties I care about most: durable state, deterministic recovery, reproducible evals, and cost under control. It also makes deletion, correction, and re-indexing tractable. I can remove a derived representation and rebuild it from its source record. I cannot safely do that when the only record is an overwritten summary or an opaque vector payload.

Memory is especially relevant in a production retrieval-augmented generation pipeline. I use the same discipline described in my production RAG systems design: source data remains authoritative, retrieval artifacts are explicitly versioned, and generated output never becomes evidence merely because it was written back into storage.

I do not assume that a larger context window solves memory. A larger window changes how much material I can present to a model in one request. It does not provide record provenance, retention rules, correction history, or deterministic reconstruction. Those are storage and data-model responsibilities.

What fails in production

The most damaging failure mode I have seen is conflating the event with its interpretation. Consider a user message saying, “Use the revised retention policy.” An episode should preserve the raw message, its timestamp, its tenant and conversation scope, and the policy version that was available at the time. A semantic record may later state that a particular policy is preferred. That statement needs a pointer back to the episode, an extraction version, a confidence or review state, and a validity interval. Without those fields, a later correction becomes an unexplained overwrite.

Another failure is using the vector store as the system of record. An embedding is a derived artifact. It depends on the source text, chunk boundaries, normalization, embedding model, model configuration, and sometimes provider-side behavior. If any of those change, retrieval can change even when the human-readable content did not. I store those inputs with the derived record so that I can identify exactly what must be rebuilt.

I also avoid a single undifferentiated “memory” table. It tends to accumulate incompatible objects: raw turns, extracted facts, summaries, tool outputs, and embeddings. Retention and access-control requirements are then difficult to apply consistently. An ephemeral tool result may deserve a short time-to-live, while a user-approved preference may require a durable audit trail.

Compression creates a quieter failure. Summaries reduce retrieval cost, but each summary discards distinctions. If I repeatedly summarize summaries, the system can preserve a fluent narrative while losing dates, exceptions, uncertainty, and source boundaries. I treat compression as a derived view with a declared input set and a versioned prompt or algorithm. I retain the source episodes unless policy requires their removal.

Finally, I have learned not to evaluate memory only through subjective chat quality. A model can produce a helpful answer after retrieving the wrong record. That is an output-quality question, not proof that retrieval worked. I measure retrieval independently before I ask a model to synthesize an answer.

A durable memory design

I start with an append-only event log. Each event receives a stable identifier at ingestion. Corrections are new events that supersede an earlier record; I do not mutate the historical payload in place. The current view is materialized from the event sequence.

I use a relational store for the authoritative metadata and payload references. For a small deployment, SQLite can be a reasonable local durability layer; its write-ahead logging documentation describes concurrent readers with a writer, while also noting that there can be only one writer at a time in WAL mode. That constraint affects my ingestion design: I batch writes, keep transactions short, and avoid treating a local database as an unbounded multi-writer queue.

A simplified record shape looks like this:

{
  "event_id": "evt_01J7X6A7YV9ZX",
  "tenant_id": "tenant_42",
  "scope": {
    "user_id": "user_18",
    "conversation_id": "conv_993"
  },
  "kind": "user_message",
  "occurred_at": "2025-03-08T10:14:21Z",
  "payload_ref": "blob://events/evt_01J7X6A7YV9ZX.json",
  "content_sha256": "a2d4...",
  "schema_version": 3,
  "ingest_run_id": "run_01J7X68N",
  "supersedes": null,
  "retention_class": "standard"
}

I use the event identifier as the anchor for every later artifact. A chunk references one or more event identifiers and records its character or token boundaries. An embedding references the chunk and records the embedding model identifier. A semantic assertion references the events and chunks from which it was extracted. A summary references the exact ordered list of inputs it compressed.

This lineage is more valuable than an informal “created at” field. It makes recovery a graph traversal: starting from an artifact, I can reach the source events; starting from a source event, I can identify every artifact affected by deletion or correction.

For semantic recall, I separate candidate generation from final context assembly. Candidate generation can combine lexical search, vector similarity, recency, scope filters, and explicit entity links. I keep the methods distinct in logs even if they are later fused into one ranked list. That allows me to learn whether a memory was found because of exact wording, semantic similarity, a conversation-local constraint, or a direct relation.

I use strict filtering before ranking. Tenant isolation, user scope, deletion status, retention status, and time bounds are eligibility conditions, not soft relevance features. A highly similar record from the wrong scope is not a near miss; it is an authorization defect.

For compression, I maintain several levels rather than one canonical summary:

1. Episode summaries describe a bounded interaction and retain links to all source events.
2. Topic summaries combine approved episode summaries for a named topic and time interval.
3. Working-memory briefs are request-specific views generated from selected evidence.

Only the first two are persisted, and both are reconstructible. I consider the working-memory brief disposable. Its role is to make one model invocation efficient, not to become long-term truth.

I also version deterministic transforms. Where I need a stable content key, I serialize normalized structured data according to a defined canonicalization scheme before hashing. RFC 8785 specifies a JSON canonicalization scheme intended to produce deterministic JSON representations suitable for cryptographic operations such as hashing. I do not claim that canonical JSON solves semantic equivalence; it only prevents insignificant serialization differences from generating needless duplicate work.

import hashlib
import json


def stable_key(document: dict) -> str:
    canonical = json.dumps(
        document,
        ensure_ascii=False,
        sort_keys=True,
        separators=(",", ":"),
    )
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

The function above is appropriate only if my schema defines the meaning of missing fields, arrays, number representations, and Unicode normalization. I make those decisions explicit in the ingest contract. Otherwise, a stable hash can create a false sense of determinism.

How I verify it

I verify recovery before I optimize retrieval quality. My recovery test begins with a known event fixture, runs ingestion and derivation, deletes all derived state, rebuilds it from the immutable events, and compares the resulting manifests. I expect stable identifiers, source links, transform versions, and content keys. If embeddings are not bitwise reproducible in my chosen environment, I record that fact and compare the parts that are deterministic: source membership, chunk boundaries, eligibility filters, and model configuration.

I keep an evaluation set separate from live traffic. Each case has a query, a scope, a cutoff time, expected evidence identifiers, prohibited identifiers when useful, and an explanation of why the evidence is relevant. The test object is the retrieval request, not merely the final answer.

{
  "case_id": "mem_eval_014",
  "query": "Which retention policy applies to export logs?",
  "scope": { "tenant_id": "tenant_42", "user_id": "user_18" },
  "as_of": "2025-03-01T00:00:00Z",
  "relevant_event_ids": ["evt_policy_07", "evt_decision_12"],
  "forbidden_event_ids": ["evt_other_tenant_03"],
  "notes": "The later approval supersedes the draft."
}

For each run, I save the complete retrieval manifest: evaluator version, corpus snapshot identifier, query normalization version, filters, candidate lists by retriever, ranking inputs, selected context, and random seed where one exists. I do not rely on a dashboard that only retains aggregate scores. A score regression without a replayable manifest is difficult to diagnose.

I calculate simple measures first. Recall at a fixed cutoff tells me whether expected evidence appears in the candidate set. Precision at that cutoff tells me how much unrelated material I am asking downstream components to inspect. I also track a policy metric: the rate at which forbidden records are returned. That metric has a different severity from ordinary relevance error.

Then I inspect slices that represent operational risk: newly ingested records, old records, long conversations, corrected facts, multilingual content when applicable, records near retention boundaries, and queries with ambiguous entity names. Aggregate metrics can conceal a regression concentrated in one of those slices.

Cost control is part of the same evaluation loop. I measure records scanned, candidates produced, reranker inputs, context bytes, and derivation work per newly ingested event. I prefer staged retrieval: inexpensive filters and sparse retrieval narrow the candidate set before more expensive embedding or model work. The exact thresholds are workload decisions, so I tune them against the fixed evaluation corpus rather than adopting a universal number.

What I would not do again

I would not persist every model-generated thought as memory. Generated text can be a useful working artifact, but storing it as a fact without provenance creates a feedback loop in which a prior inference looks like source evidence. I persist model outputs only when the product requires an audit record, and I label them as generated artifacts with their inputs and model configuration.

I would not delete raw episodes immediately after generating summaries. If storage policy permits retention, source episodes are what let me repair a defective summarizer, rebuild a changed chunking strategy, or answer a dispute about provenance. If policy requires deletion, I propagate that deletion through the lineage graph and accept that downstream artifacts must be removed or rebuilt from the remaining permitted data.

I would not hide retrieval behind one convenience method such as memory.search(query). That interface is useful at the call site, but the implementation must expose its filters, sources, versions, and ranking decisions through logs and evaluation manifests. Observability is not optional when the system is expected to remember across time.

Most importantly, I would not describe memory as a feature of the model alone. In the systems I build, memory is durable data with explicit lifecycle rules, derived indexes that can be discarded and regenerated, and retrieval behavior that can be replayed. That framing has made failures narrower, recovery more predictable, and operating cost easier to reason about.

FAQ

Why should events and derived memory artifacts be separated?

I keep immutable events separate from chunks, embeddings, summaries, and semantic assertions because derived representations can be removed and rebuilt from source records. This makes correction, deletion, re-indexing, and provenance inspection tractable.

Why is a vector store not a system of record?

I treat an embedding as a derived artifact because it depends on source text, chunk boundaries, normalization, model configuration, and sometimes provider behavior. I retain those inputs so I can identify what must be rebuilt when retrieval changes.

How do I make memory recovery deterministic?

I start with an append-only event log and stable ingestion identifiers. I version transforms and preserve lineage from every artifact to its source events. I then delete derived state, rebuild it, and compare manifests for deterministic fields.

What should a memory retrieval evaluation include?

I evaluate the retrieval request using a separate fixed set of queries, scopes, cutoff times, expected evidence, and useful forbidden records. For every run, I save filters, candidate lists, ranking inputs, selected context, versions, and any random seed.

How do I control retrieval cost without weakening policy filters?

I apply tenant, user, deletion, retention, and time constraints before ranking because they are eligibility conditions. I then use staged retrieval, narrowing candidates with inexpensive filters and sparse retrieval before more expensive embedding or model work.