Ask an agent that has handled hundreds of orders how many units of an item are left, and the answer is often wrong by a little. It is rarely wrong by a lot, which is why nobody notices. The totals drift.
A drifting count has several causes. Some are the model's arithmetic, some are how the number gets carried, and some are the data the agent was given. This post goes through each, then the one underneath, then the part an engine can fix.
Every cause, and what to do about it
1. The model does the arithmetic in text. It reads the old figure, applies the events since then, and writes the new figure as words. Nothing checks the sum. The paper Frontier LLMs Still Struggle with Simple Reasoning Tasks built tasks that include counting and raised the difficulty by lengthening the document. Its authors report that even thinking models consistently fail, and name errors in intermediate steps and difficulty with long contexts among the reasons. If a number can be computed in code, compute it in code.
2. The number is copied through a rewrite. If the agent keeps a notes block and rewrites it now and then, each rewrite is a chance to get a figure wrong, and a wrong figure is then the starting point for the next one. The error compounds and nothing flags it.
3. The summary dropped the number. When history is compacted, specific figures are easy to lose. In our own test on very long histories with Gemma 4 31B, 30 of the 43 questions a summarizing loop got wrong were answered with some version of "the detail isn't in my notes". The post on summarizing old context has the method and the limits of that test.
4. The input is long, and long inputs are read unevenly. Chroma's context rot report tested 18 models and found that performance degraded as input length increased across its experiments. Lost in the Middle found that accuracy drops when the relevant text sits in the middle of a long input rather than at the start or end. A count scattered across hundreds of events is that situation.
5. The agent counted only what it saw. If a tool returns one page of results, or the prompt holds the newest events and not the oldest, the total is a total over part of the data. No amount of care in the model fixes a number that was built from a partial view. Page through the whole set, or have the system of record return the total.
6. Two sources disagree. A stale figure in the notes and a fresh figure in a tool result can both be in the prompt. The model picks one. Decide which source wins and make that rule part of the design, not something the model guesses.
What you can do without us
Don't let the model do the bookkeeping. Keep totals in a table, update them in code from the events, and put the current value into the prompt. The model then reads the number and doesn't recompute it. The same goes for anything with a definite right answer: open orders, items shipped, money spent. Causes 5 and 6 need this plus a clear rule about which source is the truth.
The cause underneath
Causes 1 to 4 share a root. A running total is state, and when the agent keeps its state in notes, the state is stored as prose that a model rewrites. Every rewrite is a fresh chance to lose it, and the longer the run the more rewrites there are. Better prompts and bigger windows may make a single rewrite less likely to fail. They don't change the fact that the number only exists as something the model re-emits.
What an engine can fix
That is the part we build. In the engine, the agent states each change ("stock of S10 down 12") and code does the arithmetic. The totals are kept by the engine and go into every rebuilt prompt, so the model doesn't have to carry a number across a rewrite, and the engine writes the carry-over report at each rebuild without a model call. It does not fix causes 5 and 6. If the agent never sees an event, or the sources disagree, the engine can't know.
What we measured
In our own 12-hour shop run in October, the reports the model wrote at each rebuild lost track of running stock counts, and the agent answered 5 of 28 stock questions right. The known issues page lists the rows.
We ran the same 900-event shop shift twice on Claude Haiku 5.5, back to back, on the hosted engine (release 0.51.0). One run had the engine keeping the counts and one had the model keeping them, and everything else was identical, including a guard that refuses a second refund for the same order. In the second, the model writes a report at each rebuild and carries the numbers in it, as in October.
- Stock questions answered right: 22 of 28 with the engine keeping the counts, 4 of 28 with the model writing them.
- Callback questions about facts stated earlier in the shift: 91 of 91 on, 86 of 91 off. The 5 misses were all in the mid, far and very far buckets.
- Model spend for the shift: $0.41 on, $0.52 off.
Six of the 28 stock answers are still wrong with the engine keeping the counts. Each is short by exactly 50 units, which points to one 50-unit change missed or miscounted and carried forward. We haven't traced it.
Every graded event of both runs, the run totals and the refund log are downloadable rows you can recount with the recount script.
What we haven't shown
- One run per setup, on a scenario we generated, with one model. Haiku isn't deterministic, so 22 against 4 is a strong signal and not an effect size.
- Our engine against its own older setup. We have not put an agent built on another SDK through this scenario, and a developer who keeps counts in code in their own loop may well do much better than the model-written setup here.
- The engine has changed since release 0.51.0, and we haven't rerun this on later builds.
- Stock answers count exact matches only.
Sources and method: 11 hours 59 minutes on shift: what the agent remembered.