Multi-Model and Multi-Modal: Where the Memory Layer Has to Change, and Where It Doesn't
Two of the four criteria from this series’ opening article remain: multi-model routing and multi-modal memory. They get one article together because the research on each turns out to be lopsided in a genuinely useful way — one of them is close to a non-issue for the kind of text-first memory layer this series has been designing across the previous six articles, and the other is a real, hard, unsolved problem that deserves to be named as exactly that rather than glossed over with a diagram of a “multi-modal embedding layer” that doesn’t correspond to anything working anywhere.
Multi-model routing barely touches the memory layer
The instinct, going in, was that supporting several different language models — a frontier model for hard reasoning, a smaller local model for cheap bulk work, maybe a fine-tuned specialist for one narrow task — would require real architectural accommodation in whatever stores the shared memory those models read and write. Checked directly against how real systems actually handle this, that instinct turns out to be mostly wrong, and it’s worth understanding exactly why.
Anthropic’s own published account of its multi-agent research system documents genuine model heterogeneity directly: a system with a more capable model as the lead agent and smaller models as subagents outperformed a single, larger-model-only configuration by over ninety percent on their internal benchmark. The same writeup documents memory persistence in the same system — subagents saving their plan to memory so it survives context-window truncation. Read carefully, though, the two never get tied together: memory persistence there is motivated entirely by context-window management, not by anything related to which model is reading or writing it. LangGraph’s persistence documentation describes its Store and checkpointer objects as plain state, with no model-specific serialization logic anywhere in the design. Mem0’s own writeup on multi-agent memory architecture organizes the whole design around scoping dimensions — which user, which agent, which run, which application a memory belongs to — never around which model produced or consumes it, and states plainly that the same memory layer works identically across LangGraph, CrewAI, Strands, and AutoGen precisely because the store is text and text doesn’t care which model wrote it.
There’s exactly one documented exception, and it’s narrow and specific enough to prove the general rule rather than undermine it. Letta’s own documentation states that multi-modal capability depends entirely on the underlying model, with different providers supporting different tiers of image input — and then documents the fallback for providers that don’t support images at all: the image still appears in context, but rendered as a text message telling the agent an image exists, rather than the image itself. That’s real model-heterogeneity forcing a decision, but notice exactly where it happens: at the point a memory item gets rendered for a specific model to consume, not in how the item gets stored. It’s a read-time compatibility shim sitting on top of storage, not a change to storage itself.
The conclusion generalizes cleanly to this series’ own design: for a memory layer that stores structured text records — the shape every article so far has assumed, whether the writer is an agent, a regression suite, or a human — multi-model routing requires essentially no architectural accommodation. Text is the one representation every model, regardless of provider or size, reads identically. The routing decision (which model handles a given query) can live entirely in a layer above the memory store, informed by whatever signals make sense — cost, latency, task difficulty — without that decision needing to know or care anything about how the underlying memory is organized. Of the four original criteria, this is the one that turns out to be close to free, once the memory layer itself is designed right for concurrency, invalidation, and telemetry.
That routing layer has an obvious, concrete source of signal already sitting in this series’ own design, worth naming rather than leaving abstract: the writer-type field from the writers-beyond-agents article. A regression write is a structured fact with no ambiguity in it at all — recording it needs no model whatsoever, let alone a specific one, since nothing about “test 042 failed at commit a3f8c19” requires language understanding to store correctly. A bucketing write, deciding whether a new failure matches a known signature, is exactly the kind of narrow, well-bounded classification task a small, cheap model handles as well as an expensive one. An agent’s open-ended judgment call about whether a newly observed pattern constitutes a real architectural risk is the case that actually benefits from a frontier model’s reasoning depth. None of this routing logic needs to touch the memory layer’s schema at all — it’s a property of the task generating the write, which the writer-type field already distinguishes for other reasons, not a property the storage layer needs any new awareness of to support.
Multi-modal memory is not close to free, and the evidence is unusually strong
Five independent vendors, none of them coordinating with each other, were checked specifically for how they handle non-text content in persistent memory — and all five converged on the identical workaround, which is exactly the kind of independent convergence that makes a negative finding trustworthy rather than merely an artifact of one search missing something.
Mem0’s own documentation states the pattern directly: when a user submits an image or document, Mem0 processes it to extract textual information and other pertinent details, and those details — not the original artifact — are what gets added to memory. Caption, discard the source, store the caption. Google Cloud’s Memory Bank, part of the Gemini Enterprise Agent Platform, documents the identical shape, arrived at completely independently: memories can be extracted from images, video, and audio, producing a textual memory that includes information extracted from the input. Same lossy pattern, different company, no evident coordination between them. Zep’s Graphiti engine doesn’t even reach the caption-and-discard stage — its documentation defines exactly three episode types that can enter the system at all: text, message, and json. No image or audio episode type exists anywhere in the documentation, not even flagged as a future addition. Letta supports image input in a conversation, gated by whichever model’s vision capability is in use, but its memory-block documentation is text-only throughout — there’s no mechanism for an image to become a durable, retrievable memory object the way a text fact does; image support there is a single-turn context feature, not a persistent-memory one. Anthropic’s own memory tool is fundamentally a text-file store; it can view an image file and render it back into context if an agent already knows the file’s path, but nothing in the tool captions, indexes, embeds, or makes an image searchable by its content the way path-addressed text already is. Knowing where to look is the entire capability; there’s no mechanism for discovering a relevant image without already knowing it exists.
No system checked here stores a non-text memory object as itself — indexed, embedded, and retrievable by its actual content, the way a text fact is. Every single one either doesn’t let the modality into persistent memory at all, or converts it one-way into a caption and throws the source away. Given that these are five non-cooperating vendors, competing directly with each other in most cases, the convergence on one specific lossy pattern is strong evidence in itself: if genuine cross-modal storage and retrieval were a solved problem, at least one of them would be advertising it as a differentiator, because it would be a real one. None do. The active research literature in this exact space is building new benchmarks specifically to measure multimodal agent memory rather than treating it as already-measured — confirmation that the field itself still considers this open, not a gap this research happened to miss.
The honest conclusion: multi-modal agent memory is not an area with promising early production results. It’s genuinely unsolved, industry-wide, as of this writing, and every production system checked has independently arrived at the same workaround rather than solving the actual problem.
What this means for a system with real multi-modal writers
This matters more for this series’ design than it would for a purely text-based system, because the writers-beyond-agents article already established that non-agent writers — a regression run, a telemetry pipeline — routinely produce non-text artifacts as a normal part of what they report. A test failure worth recording as a knowledge-base entry often comes with a waveform capture, a screenshot of a rendered diff, a log file too unstructured to usefully caption. Treating those the way every vendor surveyed here treats images — extract a caption, discard the source — throws away exactly the evidence a human debugging that failure later would want to look at directly, which is a worse outcome for this kind of system than it might be for a general-purpose personal assistant’s photo memories.
Given that nobody’s solved genuine cross-modal embedding and retrieval, proposing to solve it here would be exactly the kind of unearned claim this series has tried to avoid making anywhere else. The honest, buildable middle ground is more modest than “solve multi-modal memory”: store the artifact itself, content-addressed, alongside its text caption, rather than discarding it the way every system surveyed does. A hash-addressed blob store sitting next to the structured text records — cheap, uncontroversial infrastructure, nothing novel about it — means the caption stays the primary retrieval mechanism, exactly like every vendor surveyed here, but the original waveform or screenshot is still there to open when a human or a more capable multi-modal model actually needs to look at it directly, rather than being gone the instant the caption was generated. That’s not a research contribution. It’s the acknowledgment that “unsolved” doesn’t have to mean “actively discarding the one thing a future solution would need to work with.”
What’s left
The previous six articles have each addressed one piece of a shared-memory architecture in relative isolation: the scale ladder, four production systems’ concurrency models checked directly against their own filed bugs, writers that aren’t agents, evaporation-based invalidation, and the honest limits of telemetry-driven improvement. This article closes out the last two original criteria, with genuinely different verdicts for each — one nearly free, one still open. The final article in this series pulls all of it together into a single reference architecture, described concretely at each of the scale ladder’s three tiers that actually need a design — tier 0 has no problem to solve, so it sits out — rather than leaving the pieces as separate essays that never quite assemble into something a reader could actually build.