Maybe the agent shouldn’t own the memory.

Don’t let the metaphor lead your architecture.

We call it an agent. So we give it memory, skills, identity and tools.

But does the agent own those things because the system needs it to — or because the human metaphor made that architecture feel natural?

That question stayed with me after a lunch conversation about agent memory.

Today at EMPOWER in Zurich I met José Luis Latorre.

José was kind enough to endure a series of badly worded questions from me over lunch, and I am genuinely thankful for the time he took. I understood only part of what he was describing in the moment. The dangerous thing was that I understood enough for it to keep bothering me afterwards.

So I spent the evening looking through AgentMemory for .NET, AgentEval, and Neo4j’s NAMS / Agent Memory work around graph-native memory and skill distillation.

What I find so good about José’s work

The easy version of “agent memory” is retrieval: something happened before, we stored it, later we retrieve something vaguely relevant and put it back into the context window.

José’s AgentMemory for .NET structures experience into persistent short-, long- and reasoning memory, with provenance, temporal state, supersession and an explicit projection layer.

His AgentEval work provides an evidence-oriented evaluation and runtime-governance toolkit for agent systems.

Separately, Neo4j’s NAMS / Agent Memory work shows how graph-native memory can support the broader direction toward provenance-grounded skills and capability management.

Taken together, these architectures suggest something larger.

And then the really interesting thing happens: memory stops being only about remembering.

experience
    ↓
structured memory
    ↓
evaluation
    ↓
patterns
    ↓
capability
    ↓
changed behaviour
    ↺

A simple SKILL.md file is static prose. It tells a runtime how to behave, but it has no inherent relationship to the executions that produced it, no built-in notion of evidence, drift, failure or repair.

What I find especially interesting in the NAMS Agent Skills work is the shift in what a skill can be. A SKILL.md no longer has to be the deepest representation of capability. It can be a materialized runtime view of a capability whose grounding, review state, drift and repair live deeper in the system.

SKILL.md becomes a snapshot of a living capability.

I think “materialized capability view” is the technically cleaner phrase. In Neo4j’s NAMS Agent Skills architecture, informed by the Agent Instruction Protocol (AIP) and its structured representation of skills, the deeper capability can carry grounding, provenance, procedure structure, review state, versions, drift and repair. SKILL.md is therefore not necessarily the canonical capability. It can be one portable runtime representation of its current approved state.

This is not something AgentMemory for .NET currently implements. It is one of the architectures that, alongside José’s work, triggered the broader question in this essay.

The DeltaDB detour

This is where another idea helped me understand what I was reaching for.

The team behind Delta does something conceptually strange and useful: shared worktree state can live deeper than the local filesystem representation that editors and compilers use.

Each participant still gets an ordinary synchronized checkout. VS Code edits files. Node reads them. Tests and builds run from them. Nothing about the familiar working interface has to disappear. But those files are a materialized working view of deeper shared state.

Delta does not abolish the filesystem. It demotes it from deepest source of truth to runtime representation.

That distinction lets existing tools keep the interface they need while the canonical model moves deeper into the system. And suddenly SKILL.md looked similar to me: the file still matters to the runtime, but it no longer has to be the place where capability fundamentally lives.

The repo already points in this direction

What makes this question more interesting is that José’s actual architecture is already less agent-centric than the phrase “agent memory” suggests.

Framework-specific surfaces sit around a reusable core. José’s implementation already separates stored memory, context assembly and runtime projection. Projection is an explicit architectural layer rather than one large prompt-building function. Owner and application scope are also represented as system concerns, although isolation remains configurable rather than automatic in every deployment.

MemoryContextAssembler
        ↓
MemoryContextProjector
        ↓
MemoryContext.Projection
        ↓
ProjectionRenderer

This separation exists today for memory projection. Extending the same pattern to shared capability, write-back and governed system learning is my extrapolation.

So my question is not “why did you build it this way?” Much of the machinery is already there. His runtime integration is agent-lifecycle-centric, but the underlying architecture increasingly looks like a reusable substrate.

I am wondering what happens if the projection idea is pushed one step further: from projecting memory into a runtime to projecting a broader shared capability into different actors, interfaces and human views—and allowing execution to propose governed changes back.

Projection is only half the story

The Delta analogy is not only about reading a materialized view. A developer edits the checkout, and changes can flow back into the deeper shared state.

That return path matters for adaptive AI architecture. An actor can operate on a runtime projection and produce evidence without being allowed to mutate canonical capability directly.

A. Ephemeral

An actor changes or learns something locally. Nothing persists. The next materialization wipes it.

projection
    ↓
actor
    ↓
scratchpad
    ↓
gone

B. Proposal

This is the most interesting tier. The actor discovers a tool failure, a better procedure, a contradiction, a missing condition or a local exception. It does not commit directly.

experience
    ↓
proposal
    ↓
evaluation
    ↓
promote / reject
    ↓
canonical capability

This is where AgentEval became important to my thought experiment. AgentEval provides both development-time evaluation and, increasingly, runtime enforcement through its Gatekeeper architecture. But neither of those is the adaptive promotion loop I am describing here.

The speculative step is different: execution produces evidence and a proposed capability change; evaluation tests that proposal; and only then can governance decide whether it should become shared capability.

The loop above is a proposed architecture, not current AgentMemory, AgentEval or NAMS runtime behaviour.

C. Governed release

In regulated environments, evaluation may not be enough. Candidate capability can require human or policy approval before it becomes shared state.

Proposed system architecture — not a description of the current AgentMemory, AgentEval or NAMS learning loop.

candidate capability
    ↓
eval
    ↓
human / policy approval
    ↓
approved capability
    ↓
new runtime projections

Why write-back matters

Capture at the point of discovery

The actor is present when a failure happens. Without a write-back path, a human has to notice the discovery, understand it, and transcribe it into some other system later. Much of the signal disappears between those steps.

Negative knowledge

Systems produce enormous amounts of evidence about what failed. Runbooks rarely contain it. Organisations repeatedly relearn that a tool breaks under a particular condition or that a procedure fails for a particular case.

Provenance

A capability change should link back to the executions that motivated it. Otherwise we cannot answer: Why does this instruction exist? Does the evidence still hold?

Repair latency

One policy changes. Five copied instructions become stale. Proposal, evaluation and promotion can update one canonical capability; projections refresh from there.

Local variants without forking

A deviation can be stored together with the condition under which it applies. Every local exception does not need to become a separate copied skill.

What if context itself is virtualized?

Instead of saying the agent has memory, skills and tools, we could say: the system materializes a runtime world for this execution.

The agent experiences that runtime world as its context.

From inside the execution it may look like “my memory, my tools, my skills, my context.” Architecturally, those may be projections of deeper system state. What flows back is not an unchecked rewrite of that state, but evidence and proposals for governed promotion.

From actor-local learning to system learning

José’s memory and evaluation work made me imagine an actor-level loop like this:

actor
    ↓
experience
    ↓
memory
    ↓
eval
    ↓
improved capability
    ↺

AgentEval spans development-time evaluation and emerging runtime governance. The move from those controls to an adaptive capability-promotion loop is the architectural step I am speculating about.

Lift that loop one level up and it scales differently:

Actor
↓
Experience
↓
Memory
↓
Eval
↓
Improved capability
↺

All actor experiences
↓
Shared evaluation
↓
Shared capability substrate
↓
Scoped projections
↓
Actor A / B / C
↺

If Actor A discovers that a tool is unreliable under a specific condition, Actor B should not have to rediscover that independently. If one workflow repeatedly fails because a policy changed, every assistant should not carry its own stale version until somebody repairs it.

Maybe the scaling move is not more agents. Maybe it is moving learning from actor-local memory into shared system learning.

Three things that make this hard

System learning needs a definition of good. Organisations are often much less explicit about that than software architectures assume.

Isn’t this just RAG again?

RAG is primarily a read-time architecture.

documents
↓
retrieve
↓
LLM
↓
answer

capability
↓
projection
↓
execution
↓
evidence
↓
proposal
↓
eval
↓
capability
↺

RAG can still be part of the substrate. This is not “better RAG.” The differentiator is the closed learning loop.

RAG externalises knowledge. The interesting change begins when execution can produce governed changes back into the system.

The easy version of the move: Basel

I already accept this architecture for computation in my Basel spatial graph: the LLM should not estimate routes, travel time or transit relationships. The deterministic spatial and temporal system owns those answers. The model extracts intent and constraints, then explains a computed result.

Taken together, these architectures made me wonder whether the same move applies to learned behaviour: not only move truth out of the model, but move capability and learning out of the actor.

Stop maintaining six truths

Organisations routinely take one real process and copy it into a wiki, a runbook, a checklist, training, AI instructions and workflow configuration. Each representation becomes a separately maintained truth. Each drifts.

ONE PROCESS
    ↓
wiki · runbook · checklist · training · AI instructions · workflow configuration
    ↓
six maintained truths
    ↓
drift

What if those are not six maintained truths, but six projections of one governed capability?

Stop maintaining six truths.

Where could this actually matter?

The model becomes interesting where one changing capability has to serve many consumers, roles or runtimes. Not every chatbot needs this. A simple document Q&A system probably does not.

shared incident capability
→ on-call engineer view
→ incident commander view
→ AI runtime view
→ audit / training view

policy + procedure + provenance + eval
→ analyst runtime
→ reviewer runtime
→ compliance view
→ audit evidence

one living process
→ hiring manager checklist
→ HR workflow
→ employee onboarding view
→ AI guidance
→ governance view

deployment capability substrate
→ SKILL.md
→ developer docs
→ CI validation
→ Codex / Copilot runtime
→ visual workflow

all support experience
→ evaluation
→ shared resolution capability
→ customer assistant
→ frontline support
→ specialist / QA

approved tools + policy + data classification
+ workflows + incidents + eval
→ HR assistant
→ research assistant
→ M365 assistant
→ employee guidance
→ governance view

The ownership question

From inside the runtime:

I remember.
I have these skills.
I can use these tools.

From outside:

The system projected relevant state,
capabilities and tools
into this execution context.

Maybe agent memory and adaptive system middleware are the same architecture viewed from opposite sides of the runtime boundary.

The question I want to ask José is now the short one:

I don’t want to build intelligent little people with software attached. I want to build good software systems in which probabilistic reasoning is one capability among others.

Starting points

• José Luis Latorre — LinkedIn
• AgentMemory for .NET — GitHub
• AgentEval — GitHub
• Neo4j Agent Memory / NAMS — GitHub
• Agent Instruction Protocol — GitHub
• Delta