Three-Tier Agent Memory: Working, Episodic, and Semantic

🌐 한국어

"Let's give the agent memory" usually gets implemented the same way: dump every message into a database and pull the most recent N back into the prompt next turn. That works beautifully for about two months. Then the conversation grows, N is too small to reach last month's decision, and raising N blows out the context window and the budget with it.

The underlying problem is treating all memories as the same kind of thing. What was just said, a decision from three months ago, and the fact that "this person writes Go" are different in nature. They differ in access frequency, in the precision they need, and in the size they should occupy. So you split them into tiers.

The Three Tiers

Tier	Holds	Lifetime	Form
Working	The conversation in progress	Session	Raw messages
Episodic	What happened in past sessions	Weeks to months	Summaries
Semantic	Facts: people, projects, relationships	Permanent	A graph

The names are borrowed from cognitive psychology, but the implementation view is simple. Working stores raw, episodic stores compressed, semantic stores structured. As you move down, information density rises and size falls.

The Core Idea: Progressive Loading (L0 → L1 → L2)

Tiering alone isn't enough. The real win comes from "how much do we load every turn?" Load everything and the tiers were pointless.

So you use three loading levels.

• L0 — auto-injected (always): a handful of one-line abstracts from relevant past sessions go into the system prompt automatically. Just enough for the agent to notice "we've discussed this before." Roughly 50 tokens per entry.
• L1 — searched on demand: when the agent decides it needs more, it searches with a tool. Results come back as IDs plus short summaries, and it expands a specific one by ID when it needs the full text.
• L2 — relationship queries: when the question is about connections — "which projects has this person touched?" — it queries the knowledge graph.

The point is that only L0 costs tokens unconditionally; L1 and L2 cost only when the agent judges them necessary. Most turns don't need past memory at all.

The agent has to understand this structure to use it well, so a paragraph goes into the system prompt.

You have 3 levels of memory:

- Auto-recall (L0): items under "Memory Context" above are auto-injected hints from past sessions.

- Episodic (L1): full session summaries — find via memory_search, then memory_expand(id) for details.

- Semantic (L2): a graph of people, projects, and connections — query with knowledge_graph_search.

Before answering questions about prior work, decisions, people, or preferences, search first.

If nothing relevant is found, say so naturally without mentioning tool names.

Implementing L0 Auto-Injection

At the start of a turn, search the episodic store with the user's message and build a prompt section from the top few hyper-compressed abstracts that clear a score threshold.

func (a *autoInjector) Inject(ctx context.Context, p InjectParams) (*InjectResult, error) {

    // Skip the search entirely for trivial messages like greetings

    if isTrivialMessage(p.UserMessage) {

        return &InjectResult{}, nil

    }

    // Build the query with conversational context mixed in (explained below)

    query := buildRecallQuery(p.UserMessage, p.RecentContext)

    results, err := a.store.Search(ctx, query, p.AgentID, p.UserID, SearchOptions{

        MaxResults:   p.MaxEntries * 2, // over-fetch, then filter by threshold

        MinScore:     0.3,

        TextWeight:   0.7,  // keyword search weight

        VectorWeight: 0.3,  // semantic search weight

    })

    if err != nil || len(results) == 0 {

        return &InjectResult{}, err

    }

    var sb strings.Builder

    sb.WriteString("## Memory Context\n\nRelevant memories from past sessions (search for details):\n")

    injected := 0

    for _, r := range results {

        if injected >= p.MaxEntries || r.Abstract == "" {

            continue

        }

        sb.WriteString("- " + r.Abstract + "\n")

        injected++

    }

    return &InjectResult{Section: sb.String(), Injected: injected}, nil

}

Three design decisions are baked in here.

• A trivial-message filter — running vector search on "ok" or "thanks" wastes money, and irrelevant memories dragged in actively get in the way.
• A 7:3 keyword-to-semantic hybrid — auto-injection runs on every turn, so its latency is your response latency. Keyword search is fast and precise at catching proper nouns (people, project names), so it gets the heavier weight. For L1 search, where precision matters more, you can flip the ratio toward vectors.
• Below threshold means nothing gets injected — feed in a marginal memory and the model mistakes an unrelated past conversation for current context. Nothing beats wrong.

The Pronoun Problem — Mixing Recent Context into the Query

Run this in production and you hit a problem immediately. The user asks, "how did that turn out?" Search on that sentence alone and you get nothing, because a pronoun has no content to search on.

The fix is simple: append the last few user turns to the query.

// Concatenate recent user turns to enrich the recall query.

// Budget: 2 turns max, ~300 characters total — longer dilutes the actual question.

func buildRecentContext(history []Message) string {

    const maxTurns, maxRunes = 2, 300

    turns := make([]string, 0, maxTurns)

    // Walk backward — recent turns survive even if earlier ones were pruned

    for i := len(history) - 1; i >= 0 && len(turns) < maxTurns; i-- {

        if history[i].Role != "user" || history[i].Content == "" {

            continue

        }

        turns = append([]string{history[i].Content}, turns...) // prepend to keep order

    }

    joined := strings.Join(turns, " | ")

    // Clip by runes, not bytes — a byte-wise cut mangles Korean and Vietnamese

    // and sends invalid UTF-8 to the embedding model

    if r := []rune(joined); len(r) > maxRunes {

        joined = string(r[len(r)-maxRunes:])  // keep the most recent portion

    }

    return joined

}

Clipping by runes rather than bytes is mandatory for a multilingual service. Korean characters are three bytes in UTF-8, so a byte-wise cut slices a character in half, and broken UTF-8 reaching an embedding API produces a quietly wrong vector. Keeping the tail is deliberate too — context closest in time to the current question is what resolves the pronoun.

Move Between Tiers via Events — Never Block the Request Path

Promotion from working to episodic, and episodic to semantic, must be entirely asynchronous. Session summarization is an LLM call taking seconds; put that on the response path and the user waits for all of it.

// Event chain: session ends -> summarize -> extract entities -> dedupe

bus.Subscribe(EventSessionCompleted, episodicWorker.Handle) // conversation -> summary

bus.Subscribe(EventEpisodicCreated,  semanticWorker.Handle) // summary -> entities/relations

bus.Subscribe(EventEntityUpserted,   dedupWorker.Handle)    // merge near-duplicate entities

// Periodically prune expired episodes — memory has to forget too

go func() {

    ticker := time.NewTicker(6 * time.Hour)

    defer ticker.Stop()

    for range ticker.C {

        if n, err := store.PruneExpired(context.Background()); err == nil && n > 0 {

            slog.Info("episodic prune completed", "deleted", n)

        }

    }

}()

What each worker does:

• Episodic worker — hands session messages to an LLM for summarization. Writing a specific summarization prompt matters: "key decisions, facts learned about the user or project, tasks completed or in progress, important technical details, preferences expressed. No greetings or filler. Include entity names explicitly." That last sentence is what keeps the next stage alive.
• Semantic worker — extracts entities (people, projects, technologies) and their relationships from the summary and upserts them into the graph.
• Dedup worker — stops "Jane Smith" and "Jane Smith" from accumulating as separate nodes. Skip this and the graph is useless within months.

Build the L0 Abstract Without an LLM

Generating the one-line abstracts with an LLM too would double your cost. Extractive is plenty here — pull the first meaningful sentence out of the summary you already have.

// Extract a ~50-token abstract from a summary. No LLM call.

func generateAbstract(summary string) string {

    for _, s := range splitSentences(summary) {

        s = strings.TrimSpace(s)

        r := []rune(s)

        if len(r) < 20 {

            continue // skip very short fragments

        }

        if len(r) > 200 {

            return string(r[:200]) + "..."

        }

        return s

    }

    if r := []rune(summary); len(r) > 200 {

        return string(r[:200]) + "..."

    }

    return summary

}

It isn't sophisticated, but it's sufficient because L0's job is a relevance hint, not accurate information delivery. If the agent reads that line and decides it needs more, it drops to L1 and fetches the precise content.

What This Structure Solves

• Token cost stops scaling with conversation length. The only fixed per-turn addition is a few L0 lines.
• Old memories stay reachable. Unlike a most-recent-N window, a decision from three months ago is still findable by search.
• Multi-hop questions become answerable — "which projects has this person been involved in?" needs a graph; summary search alone can't do it.
• Forgetting is implementable. Give episodes an expiry and old small talk disappears while facts promoted to the graph survive. That resembles how human memory behaves.

Things to Watch

• Measure recall quality. Record how many entries were injected and the top score, and you can tell whether the threshold is so high nothing matches or so low noise gets through.
• Respect privacy boundaries. A memory formed in a private conversation surfacing in a group chat is an incident. Always scope searches by user and agent.
• Beware summaries of summaries. Repeated compression distorts information. Keep the original sessions separately so a summary can always be regenerated.

Summary

• Memory splits into Working (raw) → Episodic (summarized) → Semantic (graph)
• Control cost with progressive loading — L0 always, L1 and L2 on demand
• Auto-injection uses a keyword-weighted hybrid search plus a threshold cut; when marginal, inject nothing
• Resolve pronouns by mixing recent user turns into the query (clip by runes, always)
• Promote between tiers asynchronously via events — never block the response path
• The L0 abstract can be purely extractive — its role is a hint, not information transfer

The hard part of memory design isn't storage, it's forgetting. Deciding what not to load matters considerably more than deciding what to load.