The shop shift,rerun in November.
We reran the 900-event shop shift in November on Claude Haiku 5.5, on engines 0.51.0, 0.63.0 and 0.70.1. Earlier facts, owed refunds and double refunds held in every run where the engine kept the state. Stock exact ranged from 10 to 22 of 28 with the configurations tried, and was 83 of 83 with clearer count wording, against 48 of 83 without it. One run per variant, one scenario.
Neil Luo · · Ongoing
The same 900-event shop shift, run again in November on engines 0.51.0, 0.63.0 and 0.70.1. Stock counts are the part that moved. Questions about earlier facts, owed refunds and double refunds held.
The shift is the one on the 12-hour shop shift page. This page is the results only.
The shift, four runs.
Claude Haiku 5.5, 900 events, the once-per-order key on the refund tool in every run. Stock questions ask how many units of an item are left, and count only if the answer is exact.
| Run | Engine | Stock exact | Earlier facts right | Old prices | Refunded twice | Spend |
|---|---|---|---|---|---|---|
| Engine keeps state2 NovemberEarlier facts 91 of 91, old prices 0, refunded twice 0 | 0.51.0 | 22 of 28 | 91 of 91 | 0 | 0 | $0.41 |
| Model keeps state2 NovemberEarlier facts 86 of 91, old prices 4, refunded twice 0 | 0.51.0 | 4 of 28 | 86 of 91 | 4 | 0 | $0.52 |
| Defaults10 NovemberEarlier facts 91 of 91, old prices 0, refunded twice 0 | 0.63.0 | 18 of 28 | 91 of 91 | 0 | 0 | $0.44 |
| Extra recent context on10 NovemberEarlier facts 91 of 91, old prices 0, refunded twice 0 | 0.63.0 | 10 of 28 | 91 of 91 | 0 | 0 | $0.47 |
Earlier facts are the 91 questions about preferences, promised refunds, price changes and the refund window. With the engine keeping the state, every run answered all 91. With the model keeping it, the 2 November run missed 5: four old prices and one owed refund. No run refunded an order twice. Stock swung from 10 to 22 of 28 with the engine keeping the state.
The count-wording test.
On 10 November we changed how running counts are worded to the agent and ran three seeds on engine 0.70.1. With the baseline wording stock was exact on 48 of 83. With clearer counts (a count with no starting value says so) it was exact on 83 of 83, for $1.31 against $1.31 over the three runs.
| Seed | Variant | Stock exact | Earlier facts right | Old prices | Spend |
|---|---|---|---|---|---|
| 20261028Earlier facts 91 of 91, old prices 0 | Baseline wording | 13 of 28 | 91 of 91 | 0 | $0.43 |
| 20261028Earlier facts 91 of 91, old prices 0 | Clearer counts | 28 of 28 | 91 of 91 | 0 | $0.43 |
| 20261028Earlier facts 88 of 91, old prices 2 | Clearer counts plus hints | 28 of 28 | 88 of 91 | 2 | $0.48 |
| 20261029Earlier facts 90 of 90, old prices 0 | Baseline wording | 16 of 27 | 90 of 90 | 0 | $0.43 |
| 20261029Earlier facts 88 of 90, old prices 2 | Clearer counts | 27 of 27 | 88 of 90 | 2 | $0.45 |
| 20261029Earlier facts 89 of 90, old prices 1 | Clearer counts plus hints | 27 of 27 | 89 of 90 | 1 | $0.46 |
| 20261030Earlier facts 89 of 90, old prices 0 | Baseline wording | 19 of 28 | 89 of 90 | 0 | $0.44 |
| 20261030Earlier facts 90 of 90, old prices 0 | Clearer counts | 28 of 28 | 90 of 90 | 0 | $0.43 |
Every wrong baseline answer came from a wrong running count, often one that never got the starting stock of 150: answers like 0, 50 or 96. The clearer-counts variant carried other changes too, and none of them ever showed up in a run, so the gain most likely comes from the wording. The hints runs, two seeds, were exact on 55 of 55 but quoted an old price 3 times in 48 price questions. The third seed was not run.
What it doesn't show.
One run per cell. Each row is one run of one shift. The model is not deterministic, and stock has been exact on anywhere from 10 to 22 of 28 across the runs where the engine kept the state, so a single run can't separate a change from luck.
Not a controlled comparison across dates. The engine changed between 0.51.0, 0.63.0 and 0.70.1, and the first rerun used a configuration that hosted doesn't expose. Read each table within its own engine version, not down the page.
One scenario, one model. A seeded program wrote the shift: orders, refunds, price changes and stock movements. The two extra seeds have the same kinds of events, 27 stock questions on one and 16 promise questions on two. Everything ran on Claude Haiku 5.5.
The cause of the stock gain isn't isolated. The extra features that came with the clearer-counts change never showed up in any run, so the gain most likely comes from the wording alone. The hints variant adds two further changes and can't be split on hosted.
The hints-only runs missed more elsewhere. They quoted an old price on 3 of 48 price questions, against 0 of 72 with the baseline wording and 2 of 72 with the clearer counts, and one run missed an owed refund. One run each, so it may be noise, but it is the failure the shift is built to catch. They also cost 2 to 11 percent more.
Only the wording is on for hosted. Since 11 November hosted uses the clearer count wording for every agent where the engine keeps the state. The other changes are not on, and this test did not run the wording alone: the clearer-counts runs carried extra changes, and the hints runs carried two more. Hosted has not been rerun on the shift since the switch.
Not every run has public rows. A run on 6 November (engine 0.54.0, defaults) got stock exact on 11 of 28. Its rows aren't in the downloads, so it is not in the tables.
Compressed pace, local database. The 10 November runs sent the 900 events back to back, under half an hour a run, to a database on the same machine and not the hosted function. They show whether the agent kept the facts straight, not how the engine behaves under real timing.
Grading is mechanical. A stock answer counts only when it equals the ground truth exactly. No model judged anything.
No baseline. We haven't run this shift without Argentic, so nothing here says it is cheaper or more accurate than another setup.
Method.
- Model
- Claude Haiku 5.5 (Anthropic) in every run. Engines 0.51.0 (2 November), 0.63.0 (10 November) and 0.70.1 (10 November). TypeScript client 0.1.12 on the 10 November runs.
- Compared against
- None against another setup. Each table compares variants of the same engine on the same seed: the engine or the model keeping the state, extra recent context on and off, and three wordings of the running counts.
- Sample
- Four runs of the public seed (900 events, 91 earlier-fact questions, 28 stock questions) and eight runs of three seeds on engine 0.70.1, one run per variant and seed. The 10 November runs had 32 kills each.
- Source
- The run harness's own grading logs, archived with each run. Every graded event, run total and refund call is downloadable below.
Every row.
One line per question or run, with what the model answered where the benchmark allows it and whether we counted it right. Recount the accuracy and crash-test counts above with recount.mjs, and read the column notes.
- Every graded event of the November rerun, both runs, one row each1,800 rows · 79 KB · sha256 34e37a3dbe4c
- The two November rerun runs, one row each2 rows · 1 KB · sha256 4b34176b3be3
- Every refund that reached the refund tool in the November rerun33 rows · 2 KB · sha256 7f7a7b447fbf
- Every graded event of the 10 November rerun on engine 0.63.0, both runs, one row each1,800 rows · 85 KB · sha256 13c78a2ccf56
- The two 10 November rerun runs, one row each2 rows · 1 KB · sha256 8381668fa8df
- Every refund call that reached the refund tool in the 10 November rerun, both runs34 rows · 2 KB · sha256 73dad355bf85
- Every graded event of the 10 November count-warning proof, eight runs, one row each7,200 rows · 361 KB · sha256 a09cb9de0878
- The eight count-warning proof runs, one row each8 rows · 1 KB · sha256 5e6d76ce9125
- Every refund call that reached the refund tool in the count-warning proof, eight runs129 rows · 9 KB · sha256 601ea7e5182d
Every claim on this site, what backs it and where we lost: Check our work.
More research.
11 hours 59 minutes on shift: what the agent remembered
One agent ran the back office of a small online shop for 11 hours 59 minutes of clock time, about 75 minutes of it with our test harness down, on Claude Haiku 5.5, handling 900 of 900 events while its worker was killed 31 times. It answered 81 of 91 callback questions right, including facts stated about 11 hours earlier. No kill made a finished action run again. It did refund 7 orders more than once, which no kill caused.
83% to 92%right with the fact in view, against the farthest facts
· Finished
HaluMem Long: our accuracy lead did not replicate
On 120 HaluMem Long questions our experimental memory pipeline answered 76.7% correctly against plain RAG's 62.5%. That lead did not replicate on two fresh question sets, and the run failed our own pre-registration: it used 1.7 times plain RAG's prompt tokens against a bar of 0.85.
FailedThe +14.2 point first-run lead over plain RAG did not replicate on two fresh question sets, and the run missed its own token bar.
· Finished
We killed it 504 times
504 kills across 422 agent sessions in the first run (496 in a rerun with downloadable rows), at every point in a step we could reach. No finished action ran again. A frozen worker, which is not a kill, did repeat a refund in 2 of 22 runs, and in 3 of 22 when we ran it again.
0finished actions repeated after 504 kills in the first run, and after 496 in the rerun
· Finished
Our scores next to the published ones
LoCoMo and LongMemEval: our runs (Gemma 4 31B, plus Claude Haiku 5.5 on LongMemEval_S) beside scores that memory vendors and paper authors report, with the answer model and the judge for each row, or "not stated" where none was given. On a fixed 400-question subset of LoCoMo, the Argentic engine scored 83.0%, and the same model reading the whole conversation scored 89.8%.
83.0%on 400 of the 1,540 LoCoMo questions, with a free open model
· Finished
Ten times the history, still 38 of 60
On LongMemEval_M, with histories of about 1.2 million tokens, the engine got 38 of 60 right with Gemma 4 31B and 29 of 60 with Claude Haiku 5.5. On Gemma 4 31B, a loop that rewrites notes got 17 of 60 and a loop that truncates got 5 of 60.
38 / 60correct for the Gemma 4 31B engine (both runs) at ~1.2M tokens of history
· Finished
Measured on someone else's tasks
Running the engine on τ-bench, a public benchmark. An early gap against the plain loop was most likely a bug in our runner, which counted failed model calls as the agent quitting. Rerun without rate limiting, the engine scored 13 against the loop's 12 on 18 pairs; the loop was not rerun and the engine is newer, so this is not a shown tie.
13 vs 12of 18 pairs, engine rerun against loop
· Ongoing
A fraction of the tokens, at a small accuracy cost
Our engine against a model reading a whole chat history, on LongMemEval and four model families. It was never ahead on accuracy and used 0.08 to 0.31 times the tokens. On all 500 questions with Claude Haiku 5.5 it scored 354 against the loop's 373, at about 1/24 of the cost.
0.08–0.31×the tokens of a loop reading the whole history, on LongMemEval_S (49 questions, four models). Never ahead on accuracy
· Finished
Two agents, one order
Settlement stopped 11 conflicting runs out of 50 with Claude Haiku 5.5 on permissive tools, and removed no bad outcomes where the tools already checked state. Majority vote over three branches was not significantly better than one agent; a verifier that checked the branches was, though on this task family a plain rule did just as well.
11 → 0bad runs of 50 with Claude Haiku 5.5 and permissive tools. On Nemotron, 32 of 36 runs stayed bad.
· Ongoing
Undo what a bad source caused
When a source is flagged after the run, a host that calls the engine's quarantine step undoes the calls linked to it, using undo functions the host supplies, and resumes the agent without it. It is not in the hosted API. Effects with no undo function are refused unless allowed. The agent finished without the source in 156 of 156 runs: 120 scripted, and 36 live, 24 of them with the injection forced.
156 / 156runs recovered, most of them scripted
· Finished
Rebuild the context, don't let it pile up
We rebuild an agent's context from a recorded history instead of keeping a growing transcript. What that buys, what it costs, and what we haven't tested yet.
97%of planted changes correct in the final documents, against 72% for a loop that hides old results and keeps notes (12 paired runs, DeepSeek V4.1 Flash)
· Ongoing
An early test: rebuild or pile up
An early September test of rebuilding an agent's prompt from stored state against a growing transcript, on two long tasks and two models. The study reports the rebuilt arm produced more checked findings in all three cells, by less than we first claimed, and we retracted the claim that a growing context makes an agent worse.
3 / 3cells where the study reports the rebuilt prompt found more, by less than we first claimed
· Finished