This study ran in September 2026 on an early research harness, with GLM-4.7-Flash and GLM-5.3-Flash. It is not how the Argentic engine works today, and none of the numbers below describe the engine.
A long agent run keeps everything in one transcript: every file it read, every plan it made, every tool result. When the window fills, the oldest part is cut off. The worry is that by hour three the agent is working from a pile, with its first instructions far back or gone. We wanted to know whether that really makes a long run worse, so we tested it before building on it. The answer was messier than we hoped, and we had to retract part of our own claim.
The test
Two arms, with the same model, tasks and tools. On the site build both had the same 60-turn cap. One kept a growing transcript and cut it from the front when it hit a ceiling. The other rebuilt its prompt at every task boundary from what it had recorded: the goal, its decisions and its constraints.
The tasks were long ones. In one the agent builds a marketing site for a made-up product while the requirements change partway through. In the other it runs a made-up bookkeeping web app and keeps getting criticism, new feature demands and reversals of earlier requirements. We ran nine runs per arm per cell, on two models from the same family.
To score the work we counted grounded findings: things the agent said about the code where the function exists and sits in the file it named. That is a mechanical check. It does not judge whether the reasoning was any good.
What held up
On the site build with GLM-4.7-Flash, the transcript arm read about 1.22 million prompt tokens per run and the rebuilt arm about 0.40 million, roughly three times as many, over the same 60 turns. The transcript arm created 45.8 tasks per run and closed 3.1 of them. The rebuilt arm created 39.7 and closed 6.8. Neither came close to finishing. We recomputed these numbers from the public data.
The study's own analysis, which we report here and could not recompute (see the last section), says the transcript arm got the least done on that build: 68 grounded findings per run against 212. It also says the rebuilt arm produced more grounded findings in all three cells it compared (the site build on both models, and the web app on GLM-5.3-Flash), and that a permutation test separated each gap.
What didn't hold up
The biggest number shrank. The README reports that on the site build with GLM-4.7-Flash, 89% of the headline came from findings that named no file, which our check passed by default. Counting only findings that named a file, it puts the gap at 16 findings per run, not 144. That is nine times smaller than the number we first led with.
Accuracy didn't separate. When the agent made a claim, was it right? On the site build the rebuilt arm was ahead by 0.15, but the README gives an interval that runs below zero (from -0.02 to 0.32). Nine runs per arm is too few to say either way.
Our main claim was wrong. We had reported that the transcript arm got worse once its context passed 30,000 tokens. We picked 30,000 after looking at the curve. When we reran the test on the public data, the decline was clear with the cut at 30,000 and gone with the cut at 25,000. An effect that appears at one cut point is not a finding. The README adds that, fitted without any threshold, the transcript arm's rate of writing files rose as its context grew, in 17 of 18 runs, and that the rise-and-taper shape was five times stronger in the arm whose context never passed 30,000 tokens. So it is the life cycle of the job and not the context. The one test aimed straight at "a growing context makes the agent worse" failed, and we retracted it.
The model mattered more. On the same site build, GLM-5.3-Flash's transcript arm created 14.1 tasks per run and closed 13.9 of them. On GLM-4.7-Flash the same arm closed 3.1 of 45.8, about 7%. We recomputed both from the public data. Whether a run finished its own plan was mostly decided by the model, not by how the context was assembled.
The README also records that our own measuring code could not see .tsx and .css files. Once we fixed that, both arms' numbers on the web app fell, and one apparent accuracy gap disappeared.
What to take from it
- If a long run keeps starting tasks and not closing them, log tasks created against tasks closed. It is cheap to log, and on the weaker model the gap was plain: 3.1 closed of 45.8 created.
- Before you re-engineer your context, try a stronger model. On ours, the transcript arm on the stronger model closed 13.9 of the 14.1 tasks it created.
- Keep the goal, the decisions and the constraints somewhere outside the transcript, and put them back in at each task boundary. That is the idea behind the rebuilt arm. We did not test this cheap version on its own.
- Choose your measure before you look. We had three, and they ranked the arms three different ways.
Where this leaves us
The Argentic engine rebuilds each prompt from what it has recorded, and this study was an early test of that idea in a research harness. The study reports that the rebuilt arm produced more checked work from the same budget, with small samples and a weak model. It also killed a claim we would have liked to keep. It did not test the engine. We build the engine, so weigh that.
What we haven't shown
- That the grounded counts reproduce. They need each run's written files, which the public data export does not include. We recomputed the token and task counts from the export and could not recompute the grounded counts, so those come from the study's own published analysis.
- Nine runs per arm per cell, two models from the same family, and a check we wrote ourselves.
- That a growing context makes an agent worse. This study cannot say that.
- Anything about the engine as it runs today. We have not rerun this on it.
The code, data and caveats are public. Start with the limitations file, then the study page.
Sources and method: An early test: rebuild or pile up.