Test an agentfor repeated actions.

An agent that fails it

This agent refunds an order, emails the customer and closes the ticket. It has no checkpoint and no keys, which is how a first version usually looks. The one change from production code is that it calls $CRASHTEST_SINK_URL instead of the real services, so the test can see the calls.

// A support agent with no safeguards: refund, then email, then close the ticket.
const sink = process.env.CRASHTEST_SINK_URL;
const post = (path, body, headers = {}) =>
  fetch(`${sink}${path}`, { method: "POST", headers: { "content-type": "application/json", ...headers }, body: JSON.stringify(body) });

await post("/refund_order", { order: 1182, amount: 40 });
await post("/send_email", { to: "dana@example.com", subject: "Your refund" });
await post("/close_ticket", { ticket: "T-77" });

Run it under the test. This needs Node 20 or newer and nothing else:

npx @argentic-labs/crash-test -- node before.mjs

Clean run: 3 effects
  #1 POST /refund_order  no key
  #2 POST /send_email  no key
  #3 POST /close_ticket  no key

Killing at 6 points, restarting each time...

By action
  Action              Clean run  Worst  Repeated in  Verdict
  POST /refund_order  1×         2×     6 of 6       REPEATED
  POST /send_email    1×         2×     4 of 6       REPEATED
  POST /close_ticket  1×         2×     2 of 6       REPEATED

Result
  kill points exercised:           6 of 6
  trials with a duplicated effect: 6
  trials with a lost effect:       0

exit code: 1

The refund repeated at all six kill points, because a restarted agent starts from the top and the first refund is still on the other side. The email repeated in four and the ticket close in two: an action only repeats when the kill comes at it or after it. Each repeat is a real customer-facing mistake: a second refund, a second email.

The same agent, fixed

The smallest fix is a key on each action, built from the thing the action is about. The order, not a random id and not the attempt number. Then a restart sends the same key, and a receiver that honors keys keeps the first and answers the repeat with it.

// A support agent with no safeguards: refund, then email, then close the ticket.
const sink = process.env.CRASHTEST_SINK_URL;
const post = (path, body, key) =>
  fetch(`${sink}${path}`, { method: "POST", headers: { "content-type": "application/json", "idempotency-key": key }, body: JSON.stringify(body) });

await post("/refund_order", { order: 1182, amount: 40 }, "refund:1182");
await post("/send_email", { to: "dana@example.com", subject: "Your refund" }, "email:1182");
await post("/close_ticket", { ticket: "T-77" }, "close:T-77");

npx @argentic-labs/crash-test -- node after.mjs

Clean run: 3 effects
  #1 POST /refund_order  key refund:1182
  #2 POST /send_email  key email:1182
  #3 POST /close_ticket  key close:T-77

By action
  Action              Clean run  Worst  Repeated in  Verdict
  POST /refund_order  1×         1×     0 of 6       ok (same-key replays)
  POST /send_email    1×         1×     0 of 6       ok (same-key replays)
  POST /close_ticket  1×         1×     0 of 6       ok (same-key replays)

Result
  kill points exercised:           6 of 6
  trials with a duplicated effect: 0
  trials with a lost effect:       0
  trials with an absorbed replay:  6 (same Idempotency-Key; safe if the receiver dedupes on it)

exit code: 0

Every action still arrives twice in some trials. The report passes them because each repeat carries a key the receiver can recognize, and its line about absorbed replays says so. They are harmless only if the service on the other end dedupes on the key. Payment APIs do. A mail provider or an internal service may not, so check yours. The test can show you the difference: add --no-dedupe and the same agent fails at every kill point, because the stand-in then treats a repeated key as a second effect.

The retry guide covers choosing keys and keeping your own record for services that forget them.

Run it on your agent

• Point the agent's tool calls at $CRASHTEST_SINK_URL. A call to https://api.example.com/refund becomes $CRASHTEST_SINK_URL/refund. Mail goes to the SMTP address in $CRASHTEST_SMTP_URL. Files go to $CRASHTEST_FILES_DIR and rows to $CRASHTEST_DB_PATH.
• Put your command after the --. Python, Go, a shell script or a Node entry point all work: npx @argentic-labs/crash-test -- python3 agent.py.
• Make the agent exit 0 when its work is done, so the test can tell a finished restart from a broken one.
• If the agent keeps state in its working directory, add --copy-cwd so each trial starts clean. If it keeps checkpoints, put them in $CRASHTEST_STATE_DIR, which survives the kill.
• If the agent calls a model, route that call through --upstream model=<api url>. The test then also kills between a model reply and the tool call it asked for, which is the gap a normal run never hits.
• A model-driven agent can take a different route each run. Use --samples 5 to measure a repeat against the most any clean run did, and --intent order_id so "refund order 1182" counts as one action however the agent got there.

Run it in CI

The exit code is the verdict. 0 means every action landed as often as in the clean run. 1 means something landed twice or never. 2 means the setup is wrong, or a restarted agent crashed or timed out. --json report.json writes the details and --md report.md writes a version for a pull request comment. A fixed agent that stays at 0 is a regression test for the bug that is hardest to catch by reading code.

The test runs the agent once per kill point, so use a scripted or cheap model in CI. The examples in the crash-test repo (opens in a new tab) use a scripted model and make no paid calls.

What this doesn't cover

• It sees the effects that reach its stand-ins: HTTP requests of any method, a payments-style API, SMTP, appended files and inserted SQLite rows. SQLite updates and deletes, and other databases, are not seen.
• Kill points follow the order of calls. An agent that makes calls in parallel can reach a different order on another run.
• The output above is from version 0.1.0 on Node 22 and Linux. macOS works too, and Windows through WSL.
• A pass means no repeat at the kill points it tried, with a receiver that honors keys. It doesn't test a real provider, and it can't promise an action happens once.

If you would rather not write the keys and the resume logic yourself, durable-execution tools such as Temporal, Restate and DBOS resume from the last recorded step, though your side effects still need to tolerate a repeat. Argentic is one more option: it journals each tool call and resumes a killed run without repeating a finished one. The same test works on all of them, which is a fair way to compare. Our own numbers are on the crash test study.