More correct answers,at 1.7 times the tokens.
On 120 HaluMem Long questions our experimental memory pipeline answered 76.7% correctly against plain RAG's 62.5%, using 1.7 times the prompt tokens, which missed the bar we set in advance.
Neil Luo · · Finished
HaluMem tests whether a memory system stays accurate as a user's history grows. Its Long setting gives each user about a million tokens of conversation, then asks questions that each have one right answer.
On 120 Long questions, our experimental memory pipeline answered 76.7% correctly and plain RAG 62.5%. That is 14.2 points more, with a 90% interval of +5.5 to +22.4. It cost 1.7 times the prompt tokens and about twice as much a question, which missed the token bar we set before running it.
HaluMem Long, 120 questions.
Each answer is graded correct, hallucinated or omitted, and the three add to 100. Claude Haiku 5.5 answered both rows. Tokens are what the answering model was sent for one question.
| How it answers | Correct | Hallucinated | Omitted | Prompt tokens | Cost a question |
|---|---|---|---|---|---|
| Our experimental memory pipelineCorrect 76.7 · hallucinated 20.0 · omitted 3.3 · 6.3k prompt tokens · $0.00085 a question | |||||
| Plain RAG, the 16 best-matching passagesCorrect 62.5 · hallucinated 24.2 · omitted 13.3 · 3.7k prompt tokens · $0.00041 a question |
Two more rows ran. The same pipeline without its extra search step scored the same 76.7. Plain RAG with that search step scored 64.2.
We also graded every answer with a second judge, minimax-m3.1-flash. It keeps the correct-answer gain (+9.2 points, 90% interval +0.9 to +16.8). It reverses the hallucination comparison: on that judge our pipeline hallucinated 4.2 points more than plain RAG (32.5 against 28.3). So we make no claim about fewer hallucinations.
Ahead on five types, behind on one.
Percent correct on Long, by question type, 20 questions each.
HaluMem Longour pipeline vs plain RAG
76.7 v 62.5 percent correct, 120 questions
Percent correct by question type, 20 questions each
Basic fact
Dynamic update
Generalization
Memory conflict
Multi-hop
Memory boundary
Our pipelinePlain RAG
How we ran it.
We wrote down the bars and the arms before any HaluMem answer existed, then ran each arm once. Long histories run from about 150,000 to 1.5 million tokens, with a median near a million. We also ran the same 120 questions on their Medium histories, which run from about 24,000 to 390,000 tokens, so the only thing that changes between the two is length.
Claude Haiku 5.5 answered with thinking off at its default sampling, since it takes no temperature setting. gpt-oss-20b graded at temperature 0 using the benchmark's official question-answering judge prompt, unmodified. No grader output failed to parse. A second grader has not finished on HaluMem yet.
Where it fell short.
It missed our token bar. Before the run we wrote down that the pipeline should use at most 0.85 times plain RAG's prompt tokens. It used 1.70 times, at about twice the cost a question. The accuracy bars passed. The token bar failed, so by our own pre-registration the run failed.
More confident errors. Its omission rate fell from 13.3 to 3.3 because it answers where plain RAG says it doesn't know. So nearly all of its misses are confident wrong answers: 20.0 of the 23.3 points it missed. The hallucination rates of 20.0 and 24.2 are close enough that on 120 questions we can't tell them apart.
A loss on memory boundary. Questions about things the user never said: 90 against plain RAG's 95, on 20 questions.
Flat on long histories, for both. Moving the same questions from Medium to Long histories changed plain RAG's correct rate by -2.5 points (interval -9.3 to +3.5) and ours by +0.8 (-3.7 to +5.6). Both read a fixed-size set of passages, so neither is expected to fall with length. It is not a result specific to our pipeline.
The prompt and the memory are tangled. Our pipeline answers with a longer reasoning prompt than plain RAG. We have not separated how much of the gain comes from the prompt and how much from the memory.
120 questions, one run. Six types of 20 questions, from 20 users. Claude Haiku 5.5 takes no temperature setting, so it ran at default sampling and a rerun could move scores by a few points. Per-type numbers rest on 20 questions each.
Not the paper's test. The HaluMem paper used GPT-4o to answer and to grade, ran every question, and let each system use its own retrieval. We used Claude Haiku 5.5 and gpt-oss-20b, 120 questions and one retrieval stack. These numbers do not belong in the paper's table, and we make no comparison to the systems in it.
A lab pipeline. What we tested is an experimental pipeline in our lab. It is not what the product runs today.
Part of the benchmark. HaluMem also scores memory extraction and updating. Our pipeline doesn't extract memories, so those metrics don't apply and we report none.
HaluMem is by MemTensor: arXiv 2511.03506. Its data is licensed CC BY-NC-ND 4.0. We don't host or show any of it. This page has our own scores and method only, published with the authors' written permission. The same pipeline on another benchmark: BEAM.
Method.
- Model
- Claude Haiku 5.5 (Anthropic) answering, thinking off, default sampling because the API takes no temperature setting. gpt-oss-20b (OpenAI's open model) grading at temperature 0 with HaluMem's official question-answering judge prompt.
- Compared against
- Plain RAG with the 16 best-matching passages of the history, and the same with one extra search before answering that it doesn't know.
- Sample
- 120 questions on Long histories, 6 types of 20, from 20 users, and the same 120 questions on their Medium histories. One run per row, with the first grade kept.
- Source
- Lab repository: the pre-registration, written before any HaluMem answer, and the result of 2026-10-25. The repository is private. HaluMem is by MemTensor (arXiv 2511.03506), licensed CC BY-NC-ND 4.0; we publish our own scores only, with the authors' written permission.
Every claim on this site, what backs it and where we lost: Check our work.
More research.
11 hours 59 minutes on shift: what the agent remembered
One agent ran the back office of a small online shop for 11 hours 59 minutes of clock time, about 75 minutes of it with our test harness down, on Claude Haiku 5.5, handling 900 of 900 events while its worker was killed 31 times. It answered 81 of 91 callback questions right, including facts stated about 11 hours earlier. No kill made a finished action run again. It did refund 7 orders more than once, which no kill caused.
83% to 92%right with the fact in view, against the farthest facts
· Finished
We killed it 504 times
504 kills across 422 agent sessions in the first run (496 in a rerun with downloadable rows), at every point in a step we could reach. No finished action ran again. A frozen worker, which is not a kill, did repeat a refund in 2 of 22 runs, and in 3 of 22 when we ran it again.
0finished actions repeated after 504 kills in the first run, and after 496 in the rerun
· Finished
Our scores next to the published ones
LoCoMo and LongMemEval: our runs (Gemma 4 31B, plus Claude Haiku 5.5 on LongMemEval_S) beside scores that memory vendors and paper authors report, with the answer model and the judge for each row, or "not stated" where none was given. On a fixed 400-question subset of LoCoMo, the Argentic engine scored 83.0%, and the same model reading the whole conversation scored 89.8%.
83.0%on 400 of the 1,540 LoCoMo questions, with a free open model
· Finished
Ten times the history, still 38 of 60
On LongMemEval_M, with histories of about 1.2 million tokens, the engine got 38 of 60 right with Gemma 4 31B and 29 of 60 with Claude Haiku 5.5. On Gemma 4 31B, a loop that rewrites notes got 17 of 60 and a loop that truncates got 5 of 60.
38 / 60correct for the Gemma 4 31B engine (both runs) at ~1.2M tokens of history
· Finished
Measured on someone else's tasks
Running the engine on τ-bench, a public benchmark. An early gap against the plain loop was most likely a bug in our runner, which counted failed model calls as the agent quitting. Rerun without rate limiting, the engine scored 13 against the loop's 12 on 18 pairs; the loop was not rerun and the engine is newer, so this is not a shown tie.
13 vs 12of 18 pairs, engine rerun against loop
· Ongoing
A fraction of the tokens, at a small accuracy cost
Our engine against a model reading a whole chat history, on LongMemEval and four model families. It was never ahead on accuracy and used 0.08 to 0.31 times the tokens. On all 500 questions with Claude Haiku 5.5 it scored 354 against the loop's 373, at about 1/24 of the cost.
0.08–0.31×the tokens of a loop reading the whole history, on LongMemEval_S (49 questions, four models). Never ahead on accuracy
· Finished
Two agents, one order
Settlement stopped 11 conflicting runs out of 50 with Claude Haiku 5.5 on permissive tools, and removed no bad outcomes where the tools already checked state. Majority vote over three branches was not significantly better than one agent; a verifier that checked the branches was, though on this task family a plain rule did just as well.
11 → 0bad runs of 50 with Claude Haiku 5.5 and permissive tools. On Nemotron, 32 of 36 runs stayed bad.
· Ongoing
Undo what a bad source caused
When a source is flagged after the run, a host that calls the engine's quarantine step undoes the calls the journal links to it, using undo functions the host supplies, and resumes the agent without it. It is not in the hosted API. Effects with no undo function are refused unless allowed. The agent finished without the source in 156 of 156 runs: 120 scripted, and 36 live, 24 of them with the injection forced.
156 / 156runs recovered, most of them scripted
· Finished
Rebuild the context, don't let it pile up
We rebuild an agent's context from a journal instead of keeping a growing transcript. What that buys, what it costs, and what we haven't tested yet.
97%of planted changes correct in the final documents, against 72% for a loop that hides old results and keeps notes (12 paired runs, DeepSeek V4.1 Flash)
· Ongoing