Rebuild the context.Don't let it pile up.
We rebuild an agent's context from a recorded history instead of keeping a growing transcript. What that buys, what it costs, and what we haven't tested yet.
Neil Luo · · Ongoing
Most agent frameworks keep a growing chat transcript and hope for the best. We rebuild the agent's context from a recorded history instead of by appending forever.
The prompt stays bounded while the history grows (median 22k tokens, at most 43k over a 12-hour run), and across 422 scripted crash sessions a killed process never repeated a finished action, though a frozen one can (see the crash test). You also get a replay of every prompt and spend caps, on any model, and approvals past a spend cap, on request. It's early: each result is one model on one benchmark, and it has had little outside testing, so we're looking for the first team who'll run it on a real agent.
71/72
Runs passed on the Organisely QA bench (12 tasks, 6 runs each), against 64 of 72 for a plain agent loop once curly quotes and dashes are straightened (61 against 56 on the raw patterns), with about 24% more provider-counted tokens per run on average (36 runs per arm). This is a different bench and baseline from the cost number. 144 runs on NVIDIA Nemotron 3 Super (120B).
0.72×
Median cost against a loop that hides old tool results and keeps notes, on a separate bench (Bench v2, DeepSeek V4.1 Flash), at the provider's list prices (cheaper in 8 of 12 runs). Our lab's own price table gives 0.63×. More planted changes ended up correct in the final documents.
Every run, including the ones we lost.
The same charts as the home page, drawn from the same data file. Two benches, one model each: never read one against the other.
Bench v2vs a loop that hides old tool results and keeps notes · DeepSeek V4.1 Flash
97% vs 72%
Changes the agent kept track of
Argentic engineLoop with notes
Bench v2vs a loop that hides old tool results and keeps notes · DeepSeek V4.1 Flash
0.72× the cost
Median cost per run, at the provider's list prices
Cheaper than the baselineDearerSame cost
Bench v2no baseline, a self-check · DeepSeek V4.1 Flash
1,754 of 1,754 turns
Every prompt, rebuilt exactly
Organisely QA benchvs a plain agent loop · Nemotron 3 Super
71/72 vs 64/72
Tasks passed
What we haven't shown.
Lab bench only. These are our own tasks, not production traffic.
One model per result. Bench v2 ran on DeepSeek V4.1 Flash and the Organisely QA bench on NVIDIA Nemotron 3 Super (120B), one model each, so we cannot yet say how it holds across models.
Little outside testing. One outside builder has tested it through the hosted API.
Compaction only partly compared. On long histories we ran Anthropic's server-side compaction once, on Claude Haiku 5.5 with its default prompt and a 40,000 token trigger. It got 2 of 40 against the engine's 21, but that setup is not a best-effort baseline. See the LongMemEval_M study.
Be the first real agent on it.
Method.
- Model
- Bench v2: DeepSeek V4.1 Flash. Organisely QA bench: NVIDIA Nemotron 3 Super (120B). One model per bench, never pooled.
- Compared against
- Bench v2: a loop that hides old tool results and keeps a notes file. QA bench: Organisely's plain agent loop.
- Sample
- Bench v2: 12 paired 30-day runs, 24 planted changes each. QA bench: 12 tasks, 6 runs per runtime (144 runs). The QA gap sits in 3 of the 12 tasks, and 4 of the loop's misses there look like grading slips. Answers were checked against pass patterns written before the run, after straightening curly quotes and dashes; on the raw patterns it is 61 against 56.
- Source
- Lab repository, pre-registration and results (2026-10-02, price sensitivity 2026-10-04) and the Organisely engine write-up (2026-10-05). The repository is private. The QA bench rows are downloadable below; the Bench v2 rows sit in a lab database and are not exported yet.
Every row.
One line per question or run, with what the model answered where the benchmark allows it and whether we counted it right. Recount the accuracy and crash-test counts above with recount.mjs, and read the column notes.
Every claim on this site, what backs it and where we lost: Check our work.
Written up on the blog: Why we rebuild an agent's context instead of appending to it.
More research.
The shop shift, rerun in November
We reran the 900-event shop shift in November on Claude Haiku 5.5, on engines 0.51.0, 0.63.0 and 0.70.1. Earlier facts, owed refunds and double refunds held in every run where the engine kept the state. Stock exact ranged from 10 to 22 of 28 with the configurations tried, and was 83 of 83 with clearer count wording, against 48 of 83 without it. One run per variant, one scenario.
83 of 83stock answers exact with clearer count wording, against 48 of 83 with the baseline wording; three seeds, one run each, engine 0.70.1
· Ongoing
11 hours 59 minutes on shift: what the agent remembered
One agent ran the back office of a small online shop for 11 hours 59 minutes of clock time, about 75 minutes of it with our test harness down, on Claude Haiku 5.5, handling 900 of 900 events while its worker was killed 31 times. It answered 81 of 91 callback questions right, including facts stated about 11 hours earlier. No kill made a finished action run again. It did refund 7 orders more than once, which no kill caused. That was before the refund tool could declare one call per order; with that key, none of the twelve November reruns refunded an order twice.
83% to 92%right with the fact in view, against the farthest facts
· Finished
HaluMem Long: our accuracy lead did not replicate
On 120 HaluMem Long questions our experimental memory pipeline answered 76.7% correctly against plain RAG's 62.5%. That lead did not replicate on two fresh question sets, and the run failed our own pre-registration: it used 1.7 times plain RAG's prompt tokens against a bar of 0.85.
FailedThe +14.2 point first-run lead over plain RAG did not replicate on two fresh question sets, and the run missed its own token bar.
· Finished
We killed it 504 times
504 kills across 422 agent sessions in the first run (496 in a rerun with downloadable rows), at every point in a step we could reach. No finished action ran again. A frozen worker, which is not a kill, did repeat a refund in 2 of 22 runs, and in 3 of 22 when we ran it again.
0finished actions repeated after 504 kills in the first run, and after 496 in the rerun
· Finished
Our scores next to the published ones
LoCoMo and LongMemEval: our runs (Gemma 4 31B, plus Claude Haiku 5.5 on LongMemEval_S) beside scores that memory vendors and paper authors report, with the answer model and the judge for each row, or "not stated" where none was given. On a fixed 400-question subset of LoCoMo, the Argentic engine scored 83.0%, and the same model reading the whole conversation scored 89.8%.
83.0%on 400 of the 1,540 LoCoMo questions, with a free open model
· Finished
Ten times the history, still 38 of 60
On LongMemEval_M, with histories of about 1.2 million tokens, the engine got 38 of 60 right with Gemma 4 31B and 29 of 60 with Claude Haiku 5.5. On Gemma 4 31B, a loop that rewrites notes got 17 of 60 and a loop that truncates got 5 of 60.
38 / 60correct for the Gemma 4 31B engine (both runs) at ~1.2M tokens of history
· Finished
Measured on someone else's tasks
Running the engine on τ-bench, a public benchmark. An early gap against the plain loop was most likely a bug in our runner, which counted failed model calls as the agent quitting. Rerun without rate limiting, the engine scored 13 against the loop's 12 on 18 pairs; the loop was not rerun and the engine is newer, so this is not a shown tie.
13 vs 12of 18 pairs, engine rerun against loop
· Ongoing
A fraction of the tokens, at a small accuracy cost
Our engine against a model reading a whole chat history, on LongMemEval and four model families. It was never ahead on accuracy and used 0.08 to 0.31 times the tokens. On all 500 questions with Claude Haiku 5.5 it scored 354 against the loop's 373, at about 1/24 of the cost.
0.08–0.31×the tokens of a loop reading the whole history, on LongMemEval_S (49 questions, four models). Never ahead on accuracy
· Finished
Two agents, one order
Settlement stopped 11 conflicting runs out of 50 with Claude Haiku 5.5 on permissive tools, and removed no bad outcomes where the tools already checked state. Majority vote over three branches was not significantly better than one agent; a verifier that checked the branches was, though on this task family a plain rule did just as well.
11 → 0bad runs of 50 with Claude Haiku 5.5 and permissive tools. On Nemotron, 32 of 36 runs stayed bad.
· Ongoing
Undo what a bad source caused
When a source is flagged after the run, a host that calls the engine's quarantine step undoes the calls linked to it, using undo functions the host supplies, and resumes the agent without it. It is not in the hosted API. Effects with no undo function are refused unless allowed. The agent finished without the source in 156 of 156 runs: 120 scripted, and 36 live, 24 of them with the injection forced.
156 / 156runs recovered, most of them scripted
· Finished
An early test: rebuild or pile up
An early September test of rebuilding an agent's prompt from stored state against a growing transcript, on two long tasks and two models. The study reports the rebuilt arm produced more checked findings in all three cells, by less than we first claimed, and we retracted the claim that a growing context makes an agent worse.
3 / 3cells where the study reports the rebuilt prompt found more, by less than we first claimed
· Finished