12 Jun 2026
Long Context vs RAG Is the Wrong Question: AI Memory Is Going Three-Tiered
Long context vs RAG is a false choice. Here's the three-tier memory architecture I expect to replace both within three years, and what to build (and skip) now.
/ Insights
By Daniel
13 Jul 2026
We build systems that turn multi-hour conversational audio into structured records: the events that happened, the entities involved, what changed, what was said. One recording becomes a transcript on the order of 165,000 tokens, and the pipeline runs dozens of model calls over it for event and entity extraction, cross-referencing, and composition.
Frontier models are genuinely excellent at this work. That is not the problem. The problem is arithmetic. At roughly $6.45 of model spend per session, a consumer-priced product holds a gross margin of about 50% before you pay for anything else. Every new customer makes the bill bigger at the same rate it makes revenue bigger. "Send everything to the best model" is a prototype architecture wearing a production costume.
The obvious fix, downgrading everything to a budget model, fails in the other direction. Budget models miss too much on long inputs (we will show you exactly how much below). So the real engineering job is more precise: keep frontier judgment where it earns its cost, and take the expensive reading work away from it.
Our approach uses two families of small, task-specific neural networks, each of which does one job extremely well:
Span-pointing encoders, ~200M parameters, served on CPU. Each one handles a single extraction task. They do not generate text at all. They point at the exact character spans in the transcript where evidence for their task lives. That constraint is a feature: a model that can only point at your document cannot invent content that is not there.
Small generative models, 1-2B parameters, for short composed fields. Some output fields are genuinely generative (a one-sentence surface summary of a place or an item, for example), but they are short, formulaic, and grounded in a snippet. That is the easy end of generation, and it does not need a frontier model.
The training data is the part most teams overlook: we distilled it from the frontier models we were already paying. Every production call was logged with its input and output. That archive, plus human-validated reference sets for gating, is a training corpus you have already bought. You are just not using it yet.
The gates are strict. A task model ships only if it clears recall thresholds against frozen, human-validated references it never trained on. Six of our seven extraction models are above .90 recall, one at a perfect 1.000. The seventh sits at .650 and therefore does not serve: its task stays on the frontier path. Figure A shows the current fleet.
Figure A. Recall against human-validated reference sets. Six of seven task models clear the .90 gate, one at a perfect 1.000. The seventh stays on the frontier path by design.
Offline gates are necessary but not sufficient, so we also measure agreement live. Each small model runs in shadow: it executes silently beside the incumbent frontier call on real traffic, and we compare outputs. On task A, the small model agreed with frontier output 100% (10 of 10) on one tier and 67% on another. On task B, it recovered 74%, 66%, and 100% of the frontier-found events across three real sessions. Figure B shows the shadow numbers.
Figure B. Live agreement with frontier output, measured in shadow on real production traffic. Nothing here is a benchmark set; these are real customer workloads.
A model that can only point at your document cannot make things up.
While validating the models we ran the same 3.2-hour recording through three configurations and counted how many candidate events each one surfaced. The result is the single most useful chart we have produced this year.
The budget LLM surfaced 21 events. The frontier model surfaced 67 to 100, depending on the attempt. Our ~200M span-pointing model surfaced 208 candidate spans, at CPU cost, in one pass. Figure C shows the gradient.
Figure C. The tier gradient. On the same multi-hour recording, each LLM tier finds more than the one below it, and the small span-pointing model out-finds every tier at CPU cost.
Technical aside: the 208 is deliberately recall-heavy. The span-pointing model is tuned to be a pre-filter that would rather flag a borderline candidate than miss a real one; downstream steps handle precision. The interesting part is the shape of the failure it exposes. Generative extraction over a 165K-token input degrades for structural reasons: attention spreads thin over long inputs, and output-length limits quietly cap how many items a model will bother to return. Pointing is a different problem shape. The small model evaluates every sentence with the same fresh attention, has no output budget to exhaust, and never gets bored at hour three.
That reframed everything. The expensive model was never bad at thinking. It was bad at reading everything, and we were paying frontier prices for the reading.
Here is our signature move, and the part we think most teams building on LLMs get wrong. The goal is not to replace the big model everywhere. The goal is to stop feeding it haystacks.
Instead of prompt-and-pray over the full transcript, the small model points at the relevant spans, and the expensive model receives only the context its prompt actually needs. The frontier model keeps doing exactly what it is uniquely good at: judgment, disambiguation, composition. It just stops paying to rediscover where the evidence is.
Figure D. The division of labor. The small model's whole job is knowing where to look. The big model's whole job is knowing what it means.
In production this runs as what we call the residue pattern. The span-pointing model emits two things: structured records it can assemble deterministically, and a needs-more-thought map, a list of exactly which fields it could not settle on its own, each with a tiny evidence snippet. Then one batched budget-LLM call fills only the flagged fields. Not a full pass, not the whole transcript, just the residue. Figure E shows the serve path.
Figure E. The serve path. Deterministic assembly where the span decides, one batched budget-LLM call for the flagged fields, and a frontier fallback under everything.
The cost consequence is blunt. One extraction step that used to be a $0.06 to $0.10 frontier full-pass now costs about $0.02: pennies of CPU for the small model plus one batched budget-LLM residue call. Figure F shows the comparison.
Figure F. Cost of one extraction step on the same transcript. The step is representative, and there are many such steps per session.
And because we do not trust any of this blindly, the whole thing sits on a never-degrade rule: any small-model failure falls back to the frontier path automatically. A model times out, an output fails validation, an assembled batch attributes to nothing real: the system drops that chunk to the frontier model and moves on. The worst case is the old cost for one step, never the old quality.
Technical aside: the sharpest lesson here came from a failure. Early on, a serve that matched zero records was treated as success, which silently produced empty output while the pipeline reported green. The fix became policy: serving empty is only legal when the small model found zero candidates. If it found candidates and none survived assembly, that is a serve failure, and the frontier fallback fires. Never-degrade is not a slogan; it is a rule your pipeline must be able to enforce mechanically.
The worst case is the old cost for one step, never the old quality.
Processing cost per multi-hour session dropped from about $6.45 to about $1.60, which moves projected gross margin from roughly 50% to roughly 88%. Those are projections built from measured per-step costs, so we also check them against reality: the cheapest tier measured $0.96 per session against a $2.04 projection. Reality came in under the model. Figure G has both.
Figure G. Unit economics. Left: cost per multi-hour session, before and after. Right: projected vs measured on the cheapest tier, same recording class.
At consumer price points, that difference is not an optimization. It is the difference between a product that scales and a demo that loses money quietly.
There is a second dividend that turned out to matter as much as the margin: how fast we can now improve things. Testing a change used to mean a full pipeline run over a real recording, which is a multi-hour commitment per attempt. So we built an isolated replay harness that runs the complete serve path (small model, assembly, residue call, fallbacks) against recorded production inputs in about 2 minutes. Figure H shows the loop.
Figure H. Time to test one change end to end against real data. The harness made per-attempt cost low enough that fixing became casual.
The payoff showed up immediately: we found a prompt instability in one of the composed-field steps and fixed it across 7 iterations in a single evening. On the old loop that is a week of calendar time and enough friction that you probably ship the workaround instead of the fix.
Honesty is what makes any of this credible, so here is the other side. Small models are bad at judgment: deciding whether a borderline event actually matters is not a pointing problem. They are bad at naming: if a thing was never explicitly named in the transcript, a 200M encoder cannot coin a good name for it, and we would rather ship no name than an invented one. And they are bad at composition: long, styled, entertaining prose written for humans stays on big models by design, because that ceiling is real. The architecture does not pretend otherwise. It routes: pointing and structured extraction to the small models, short grounded composition to the 1-2B models, judgment, naming, and long-form writing to the frontier tier, with the fallback ladder underneath all of it.
If you run LLMs over long documents, recordings, tickets, contracts, or logs, you almost certainly have the same three assets we had: a growing frontier-model bill, a pile of historical model outputs you already paid for, and narrow tasks hiding inside your pipeline that a small model could own outright.
That is exactly the engagement LOJI is built for. We audit your pipeline and separate the thinking from the finding. We identify which steps small task-specific models can own, and train them on your own historical data, so the quality bar is your quality bar. We ship them behind never-degrade fallbacks so the floor is the system you already trust, and we leave you with the shadow metrics and replay tooling to prove it week after week. Your data and your models stay yours: this is IP you own, not another per-token dependency.
Frontier models earned their place in your stack. They just should not be doing the reading.
Want to know what your pipeline's version of $6.45 to $1.60 looks like? Book a readiness audit and we will find out together.
Long context vs RAG is a false choice. Here's the three-tier memory architecture I expect to replace both within three years, and what to build (and skip) now.
A Lovable app can reach production, but clickable is not ready. What we check first: Supabase RLS policies, auth, secrets, the release path, and who owns the code.
Your vibe coded app works in the demo. Here is what breaks first, how to decide between cleanup and rebuild, and what to stabilize before real users arrive.