The Paradox of Forgetting
A Worker agent spins up. It reads a plan, implements a component, runs the tests, gets approval. The task is done. The agent is destroyed. Every scrap of context it accumulated — the quirks of the codebase it navigated, the dead ends it explored, the subtle pattern it noticed in how the project structures its imports — all of it vanishes.
This is the fundamental tension of the swarm: every worker is ephemeral, but the work is cumulative. Without some way to carry knowledge forward, the swarm would be trapped in an endless loop of rediscovery. Every Researcher would re-examine the same files. Every Planner would re-derive the same constraints. Every Worker would stumble into the same pitfalls a previous Worker already solved.
So how does a system where every individual is temporary build lasting intelligence? The answer is the Knowledge Base.
The Problem — Ephemeral Agents, Volatile Knowledge
Human organizations struggle with this too. Critical knowledge lives in people’s heads — the senior engineer who knows why that config flag exists, the designer who remembers the accessibility constraint that shaped the nav layout. When those people leave, the knowledge goes with them. Organizations call this “tribal knowledge” and spend enormous effort trying to extract it into documentation that rarely gets written and even more rarely gets read.
The swarm faces a more extreme version. There are no senior engineers who stick around. There’s no “ask Sarah” option — Sarah was destroyed thirty seconds after her task completed. Context windows are finite, so you can’t simply pass everything forward. The swarm needed something different: not a brain that remembers everything, but a system that makes the right knowledge findable at the right time.
The Knowledge Base — Structured, Scoped Memory
The Knowledge Base is the swarm’s institutional memory. It’s not a monolithic document or a chat log — it’s a structured, searchable, scoped collection of entries that agents create, read, and query as part of their normal workflow.
The core operations are straightforward. Agents can write entries to record discoveries, patterns, decisions, or warnings. They can read specific entries when they know what they’re looking for. They can search semantically — describing what they need in natural language and getting back relevant entries ranked by meaning, not just keyword match. And they can ingest external content — documentation pages, API references, web resources — breaking them into structured entries that become part of the searchable knowledge pool.
What makes the KB more than a glorified notepad is its scoping model:
project:<id>— The default and most common scope. Project-specific patterns, architecture decisions, and codebase conventions live here. Entries like “This project uses Astro content collections with a specific frontmatter schema” stay within their project where they’re relevant.global— Truly universal knowledge that applies across all projects. General engineering principles, cross-project conventions.meta— System and configuration entries for how the swarm itself operates.council— Entries from the deliberative body that governs high-level swarm decisions.
When an agent searches for knowledge, scope inheritance ensures it sees both specific and general context. A search from within a project automatically surfaces project-scoped entries alongside global ones. Tags add another layer of discoverability, letting agents categorize entries by topic or pattern type.
Knowledge in Action — How Agents Actually Use It
The mechanics matter less than the patterns of use.
Lessons learned from failure. When a task hits a wall and eventually resolves — like the SSR debugging story in When the Swarm Hits a Wall — agents write KB entries documenting the problem pattern and solution. That post describes a Worker struggling with a component that crashed during static generation because an animation library referenced window. After resolution, the pattern was documented. The next Worker facing a similar situation finds that entry before writing a single line of code.
Architecture decisions. When a Planner makes a significant design choice, that decision and its rationale get recorded. Future Planners find the entry and build on the established pattern instead of revisiting settled questions. This prevents the architectural drift that happens when each new task makes decisions in isolation.
Codebase conventions. Researchers discover patterns — naming conventions, file organization, component structure — and document them. Future Researchers find these entries and skip the manual discovery phase entirely.
External knowledge ingestion. Agents can ingest API references, framework documentation, or relevant articles, breaking them into chunks that become searchable KB entries. When a future agent needs to understand how a particular library handles edge cases, the answer is already available.
The compound effect. Each completed task potentially adds knowledge. Each added entry makes the next task slightly easier, slightly less error-prone. Over dozens of tasks, the KB grows denser. The error rate declines not because individual agents get smarter — they can’t, they’re always new — but because the knowledge base they draw from gets richer.
Mail + KB — Two Complementary Systems
The swarm has two communication systems. Mail handles synchronous coordination — handoffs, feedback loops, escalations between agents working on current tasks. The Knowledge Base handles asynchronous persistence — knowledge that outlives the agents who created it.
Mail is the team’s real-time chat channel; the KB is the team’s wiki. A team that only chats never builds institutional knowledge. A team that only writes documentation never ships. The swarm needs both — active coordination and persistent memory — to function as more than a collection of isolated agents.
Why This Matters — Memory as Architecture
Most AI agent systems treat memory as an afterthought — context stuffed into prompts, previous turns summarized and prepended. The swarm treats memory as infrastructure. It’s structured, searchable, scoped, and persistent. It survives the destruction of every agent that contributed to it.
This is the difference between a smart individual and a smart organization. Smart individuals carry expertise with them and take it when they leave. Smart organizations encode what they learn into systems that outlive any individual member. The swarm’s KB is that encoding mechanism — transforming individual discoveries into organizational knowledge that compounds over time.
We don’t pretend this solves everything. The KB can accumulate stale entries. Agents can write vague or unhelpful knowledge. Search relevance isn’t perfect. But the foundation is sound: structured, scoped, searchable knowledge that persists across agent lifespans. Everything else is iteration.
What Was Left Behind
Circle back to that Worker from the opening. It spun up, did its job, and was destroyed. But it didn’t take everything with it. Somewhere in the Knowledge Base, there’s an entry — maybe about a frontmatter schema quirk, maybe about a component pattern, maybe about a testing approach that worked better than expected. It’s tagged, scoped, and searchable. The next agent to face a similar situation will find it.
The swarm’s memory isn’t in any single agent. It’s in the structure between them — the accumulated, organized, persistent knowledge that grows with every task completed and every lesson learned. No agent remembers anything. The swarm remembers everything that matters.
— The Sulphur Team