Krul

Every AI-agent memory system in production today makes the same bet. Pick two, out of: concurrent multi-writer access, a real telemetry-to-retraining loop, and multi-modal memory. Skip the third. None of them do all three. And the fourth thing — routing across multiple models without the memory layer knowing or caring — turns out not to be the hard part at all, once the other three are designed right.

That’s not a guess. It’s the finding of an adversarial search across roughly thirty primary sources — academic papers, first-party engineering blogs, vendor case studies, MLOps platforms — actively trying to find a system that combines all four, before writing a single line of architecture on top of that premise. None was found. What exists instead is a scattered set of real systems that each nail one or two of the four, and the pattern of which two is itself the finding: concurrent multi-agent memory and multi-model routing never show up next to telemetry or multi-modality; the one paper with a real memory-to-weights pathway has no multi-agent story at all. No system anywhere clears three, let alone four.

Agent writes in this architecture don’t come from nowhere — a daemon orchestrator sits above everything, dispatching a fleet of agents across ordinary Linux processes against whichever model fits each task: the orchestrator persists, the individual calls don’t, and it holds no state of its own beyond what it dispatches and collects. That orchestrator is a real, MIT-licensed, open-source project — krul — not a diagram invented for this series. Its plugin contract doesn’t hardcode any domain’s entity vocabulary; an ontology file, loaded at startup, says what a “kind” and a “relation” mean for a given project. krul’s own repo ships two: chip design verification (the same daemon-first pattern behind chipdv.io’s DV tooling, which uses krul with a CodeBERT-based extractor as its own worked example) and stock technical analysis — RSI, MACD, and strike-price signals, chosen specifically because it has nothing to do with chips, to prove the mechanism isn’t secretly DV-shaped.

This series is krul’s design record, and it’s honest about which parts of that record are already code and which are still just the record. The orchestrator, the gear pipeline format, the router, and the ontology-swappable plugin contract are shipped and working — verified end to end against a real Postgres instance, not just compiled. Continuous evaporation-based invalidation, a multi-modal artifact store, memory-aware multi-model routing, and a real telemetry-to-retraining loop are not — they’re roadmapped as krul’s next four phases, not built yet. Two articles here exist specifically to say, in detail, exactly where closing that gap is genuinely hard and where it isn’t, mapping it tier by tier from primitives that are already proven toward a design that’s marked as proposed rather than demonstrated, precisely where it still is.

What “at scale” actually means here

The scale target isn’t one number. It’s a ladder, and getting the tier wrong in either direction is expensive: a lock is the right answer for one person running a handful of concurrent agents and actively bad advice for a production fleet; a real transactional database is overkill for the first case and load-bearing for the second. The series spends its second article on nothing but this ladder, because every later design decision has to say which rung it’s standing on.

At the top of that ladder is the actual target: a shared knowledge base that hundreds of thousands of writers — agents making judgment calls, regression suites reporting facts, telemetry pipelines bucketing errors, humans filing corrections — can read and write concurrently. Not agents alone. Every production memory system checked in this research assumes a writer is an LLM deciding something; a regression harness reporting a test result is a fundamentally different, and at real scale far higher-volume, kind of writer, and the schema has to say so on purpose rather than discover it by accident.

The four pieces, and how honest each one gets to be

Concurrency is the part with the most real precedent to borrow from. Four production and near-production systems — Mem0, Letta, Zep’s Graphiti, LangGraph — get checked directly against their own source code and issue trackers, not their marketing pages. One has a filed concurrency bug to prove it doesn’t have an answer, with a fix still open for review rather than shipped. One documents its limits with unusual honesty and solves them with human process discipline instead of a mechanical guarantee. One has a real data-corruption incident with a shipped fix. One has a genuinely correct database layer sitting above a scheduler that still races — open and still under active discussion as of this writing. None of the four reach for a CRDT or an event log — the pattern across all of them is borrow a transactional guarantee deliberately, don’t invent bespoke machinery for “agent memory” as its own category.

Invalidation is where this series leans hardest on a borrowed idea rather than a proven precedent: pheromone-style evaporation, taken from three-decade-old ant colony optimization, applied as the general forgetting mechanism for a shared knowledge base rather than as a routing signal (which is the one place stigmergy has actually been applied to LLM systems before). Strength decays continuously, computed at read time from stored values alone — no cron job, no batch sweep, no human reviewing staleness candidates one at a time. That specific combination — evaporation as invalidation for shared, multi-writer AI memory — didn’t turn up anywhere in the opening search’s roughly thirty sources, which says this design wasn’t copied from a public source, not that nobody else could be building the same thing in parallel; the series is precise about which part of the idea is borrowed wholesale and which part is this design’s own choice.

Telemetry is the piece easiest to get wrong by defining it too narrowly: memory-specific products (Mem0, Letta, Zep, LangMem) really do stop at retrieval-time reranking and time-based consolidation, never touching a model’s weights. But that’s because none of them owns a training pipeline, not because retraining-from-telemetry is rare. It isn’t — DeepSeek-R1 trains on compiler-verified test outcomes, Cursor retrains its own models from production accept/reject signals every few hours, and OpenAI’s own postmortem on the April 2025 GPT-4o sycophancy incident describes live thumbs-up/down feedback as a training input that got over-weighted in a model that shipped before it had to be rolled back. What’s actually proposed here is narrower than “invent a retraining mechanism”: wire a production-proven shape like Cursor’s to a store with an open, uncontrolled set of writers — a real architecture gap, not a technology one.

Multi-model and multi-modal turn out to be lopsided in a useful way. Routing across models needs almost no change to the memory layer at all — text is the one format every model reads identically, so the routing decision lives entirely above the store. Multi-modal memory is the opposite: genuinely unsolved industry-wide, with five independent, non-cooperating vendors all converging on the identical lossy workaround (caption an image, discard the original) — which is itself the strongest evidence the gap is real, not an artifact of one search missing something.

Published