Today we’re sharing a preprint that’s been months in the making: “Experiential vs Synthetic Memory in Long-Running AI Agents” — now available on Zenodo (DOI: 10.5281/zenodo.18798227).
The core question is deceptively simple: Does an AI agent that accumulates real project experience outperform one given equivalent synthetic knowledge?
The answer, it turns out, is more nuanced than “yes” — and the implications could reshape how we think about onboarding AI agents onto real-world projects.
Why This Question Matters Now
The timing isn’t accidental. Just this week, Anthropic shipped Auto-Memory for Claude Code — the ability for Claude to automatically maintain a memory file across coding sessions. It’s a clear signal that the industry recognizes agent memory as a first-class problem.
But there’s a deeper question that nobody has empirically tested in the context of real software development: What kind of memory actually matters?
Not all knowledge is created equal. There’s a fundamental difference between an agent that lived through a difficult debugging session and one that was told about the same bug and its resolution. We wanted to measure that difference.
Two Types of Memory
Our paper distinguishes between two categories of agent memory:
Experiential Memory
This is knowledge accumulated through direct interaction with a project over time. Debugging episodes where the agent traced a problem through multiple layers. Architecture decisions made under real constraints. Failure logs from production incidents. User preference patterns learned through repeated collaboration.
The key characteristic: experiential memories are causal chains. They don’t just record what happened — they capture why it happened, what was tried, what failed, and what finally worked. Each memory carries the full context of its origin.
Synthetic Memory
This is equivalent knowledge assembled from external sources. Web search results about common patterns. Official documentation. LLM-generated summaries of best practices. Stack Overflow answers. Tutorial content.
The key characteristic: synthetic memories are isolated facts. They’re accurate, often high-quality, but they lack the causal structure that comes from lived experience. They tell you what to do without the why that comes from having tried alternatives.
The Experiment
We designed a controlled comparison across four conditions:
- Experiential only — Agent with accumulated real project memories, no synthetic supplements
- Synthetic only — Agent with equivalent knowledge from external sources, no project history
- Hybrid — Agent with both experiential and synthetic memory
- Baseline — Fresh agent with no memory at all
Each condition was tested across four categories of software development tasks: bug diagnosis, architecture decisions, code review, and feature implementation.
The experiment framework is deliberately practical. We’re not testing toy problems — these are tasks drawn from real codebases with real complexity.
Why This Hasn’t Been Done Before
You might assume this comparison already exists in the literature. It doesn’t — at least not in the context of real software development workflows.
Existing research on agent memory tends to fall into two camps: theoretical frameworks that describe how memory should work, and benchmarks on artificial tasks that don’t reflect the messy reality of production codebases.
What’s been missing is an empirical comparison that asks: given the same information content, does the form of that memory — experiential versus synthetic — affect agent performance on real development tasks?
That’s the gap our paper addresses.
What We Found
We’ll let the paper speak for itself on the detailed results (we encourage you to read it), but here are the key insights:
The form of memory matters as much as its content. How an agent acquired knowledge — through direct experience versus external reference — affects how effectively it can apply that knowledge to new situations.
Experiential memory shows particular advantages in tasks that require causal reasoning: diagnosing why something broke, predicting side effects of changes, and understanding the intent behind existing code.
Synthetic memory excels in tasks that require breadth of knowledge: identifying applicable patterns, suggesting libraries or tools, and referencing established best practices.
The hybrid condition suggests these aren’t competing approaches — they’re complementary layers of a complete memory system.
Implications for the Industry
If our findings hold up to further scrutiny (and we’re actively inviting that scrutiny), the implications are significant:
Memory packs become a new category of developer tools. If experiential memory demonstrably improves agent performance, then curated collections of real project experience become genuinely valuable. Not as a novelty — as infrastructure.
Agent onboarding changes fundamentally. Today, onboarding a new AI agent onto a project means starting from zero and waiting for context to build up organically. If experiential memory is transferable, onboarding becomes minutes instead of weeks.
The “fresh agent” pattern has a measurable cost. Every time you start a new session with a blank-slate agent, you’re leaving performance on the table. This isn’t just a feeling — it’s quantifiable.
Platform-locked memory is a strategic limitation. If memory is this important, locking it to a single vendor’s ecosystem isn’t just inconvenient — it’s actively harmful to developer productivity. Open, portable memory standards matter more than we thought.
How This Connects to Soul Spec
This research didn’t happen in a vacuum. It grew directly out of our work on the Soul Spec — an open specification for AI agent memory.
Soul Spec was designed around the intuition that experiential memory is fundamentally different from synthetic knowledge. The paper is our attempt to rigorously test that intuition.
The practical upshot: Soul Spec’s architecture already supports the distinction between experiential and synthetic memory. Memory packs built on Soul Spec can carry experiential knowledge — complete with causal chains and project context — in a format that’s portable, encrypted, and verifiable.
What’s Next
This is a preprint, not a final word. We’re actively looking for:
- Feedback on methodology. Did we miss something? Is there a confound we didn’t control for? We want to know.
- Replication attempts. The experiment framework is designed to be reproducible. We’d love to see others run similar comparisons.
- Contributions to the dataset. If you’re working with AI agents on real projects and willing to share anonymized memory data, that directly strengthens the research.
Try It Yourself
The paper is freely available on Zenodo. The Soul Spec documentation lives at docs.clawsouls.ai. And the broader ClawSouls platform is at clawsouls.ai.
We’re a small team, and AI has been central to how we work — not just what we build, but how we build it. This paper is one step in a longer journey toward making agent memory open, portable, and genuinely useful.
Read the paper. Try the spec. Tell us what we got wrong.
That’s how this gets better.