Agent architecture · Post 04
Memory should be a skill, not a file loaded at startup
Most agent memory behaves like luggage packed before a journey: one Markdown file is loaded by the harness, carried through every turn, and rarely opened again. Memory should behave more like recall—selective, contextual, and available at the moment it matters.
Working principle
Treat memory as a discoverable skill: keep a small registry for routing, attach explicit recall triggers to granular memories, and read the latest relevant memory at the moment of use.
Static memory is a snapshot, not a memory system
A common agent setup stores preferences, project facts, and past decisions in a Markdown file. The agent harness injects that file into the system prompt or initial context. This is convenient because it needs little infrastructure, but it makes memory behave like configuration: selected once, loaded eagerly, and fixed for the life of the thread.
Two problems follow. First, every thread pays the token cost for every loaded fact, including facts unrelated to the current task. Second, each thread begins with its own snapshot. If one thread writes an important update, other running threads continue from the version they received at startup. An agent may write memory back, yet it has little reason to read the file again before the next task needs it.
Eager loading.
Memory consumes context before relevance is known, competing with the user request and working material.
Snapshot isolation.
Parallel threads keep stale copies even after another thread records a newer decision.
Weak recall.
The system defines a write path, but rarely defines an event that causes memory to be read again.
Give each memory the shape of a skill
A skill already has the behaviour memory needs. It is named, bounded, discoverable, and loaded only under defined conditions. Instead of one large memory file, an agent can have granular memory skills such as “Project A memory,” “writing preferences,” or “deployment conventions.” Each skill states what it contains, where its current source lives, and which events should trigger recall.
For Project A, triggers might include entering its repository, seeing its name in a request, making an architectural decision, preparing a release, or resolving an ambiguity that previous work may already have settled. When a trigger matches, the agent reads the memory then—not merely the copy available when the thread started. The recalled material becomes task context, while unrelated memory remains outside the prompt.
Memory becomes useful when retrieval is part of behaviour. Storage without a recall policy is only an archive.
Start with a small memory registry
Selective recall creates a discovery problem: the agent cannot load a memory it does not know exists. The answer is a compact registry that acts as memory about memories. It contains names, scopes, short descriptions, trigger hints, locations, and freshness metadata—not the full remembered content.
The registry is the only layer suitable for routine loading. It lets the agent route a task toward relevant memory skills at low token cost. It can also support active recall: when the agent suspects prior context exists but no obvious trigger fires, it searches the registry, chooses a candidate, and retrieves that memory before deciding. The registry should stay small enough to scan and structured enough to query.
Identity.
A stable name and scope distinguish project, user, organisation, and general memories.
Triggers.
Explicit events describe when automatic recall is expected and when active lookup may help.
Freshness.
A version or updated timestamp lets the agent detect that a previously read memory may be stale.
Recall the latest state across threads
Loading memory at the moment of use changes the concurrency model. Threads no longer depend only on a startup snapshot; each can retrieve the latest shared Project A memory when a relevant event occurs. A decision recorded by one thread can therefore influence another thread that reaches its next recall point.
Dynamic retrieval does not remove coordination problems. Concurrent writers still need version checks, append-only decision records, or a merge policy so one update does not silently overwrite another. The memory skill should expose provenance and freshness, separate observed facts from inferred guidance, and make conflicting entries visible. Shared memory should become more current without pretending to be perfectly consistent.
On-demand reading improves freshness. Versioned writing protects truth. A useful memory system needs both.
Design for recall, not total prefetch
Human work offers a better metaphor than prompt preloading. We do not consciously fetch every relevant experience before beginning a task. The situation provides cues. Familiar names, places, goals, and problems activate related memories. When recall is incomplete, we pause and search: check notes, revisit a decision, or ask what happened last time.
Agent memory can follow the same rhythm without claiming to reproduce a human brain. Triggers provide associative cues. The registry supports deliberate search. Granular memory skills bound what is recalled. The working context holds what matters now. This creates a layered system in which retrieval effort grows with the task instead of making every request pay for the entire past.
Cue.
Task context activates one or more memory triggers.
Recall.
The agent reads the smallest relevant memory from its current source.
Reflect.
After meaningful work, the agent updates memory and registry metadata with provenance.
Define a minimum contract for memory skills
A memory skill should declare its scope, trigger events, retrieval location, read policy, update policy, and conflict behaviour. Its content should favour durable decisions, constraints, preferences, and unresolved questions over a transcript of everything that happened. Summaries may be useful, but provenance must remain available when a later agent needs to verify why a memory exists.
The first implementation can remain file-based. The architectural change is not a database; it is moving memory from unconditional prompt content into an explicit discovery and retrieval protocol. A registry plus granular, triggerable Markdown skills already creates selective recall. More advanced stores can follow when search, permissions, scale, or cross-device synchronisation require them.
The important shift is from “what should every thread remember at startup?” to “what should this agent recall now, why was it triggered, and is it still current?” Memory as a skill turns stored context into an active capability.