Recover a run with retry, resume, and rollback

A run can fail halfway through a hank, and the useful question is rarely "how do I restart it" but "what is the smallest move that fixes the cause without throwing away finished work?" Hankweave gives you four: retry a temporary failure, edit the instructions or environment and try again, roll back to a known-good checkpoint when file state is damaged, or start a fresh run when the persisted plan itself is the problem. A hank (the JSON program) defines the codons; a checkpoint is the file state a codon seals so recovery has somewhere safe to land.

This page walks through each move in the order you are likely to need it: automatic and manual retry, resuming an interrupted run, rollback through the control surface, the special case of a failed exhaust extension, and replaying recorded output when you want to inspect a run without executing it. It closes with a decision table for choosing between the moves and a list of recovery patterns that make failures worse.

Retry a failed codon that might work#

Retry is the right move when the failure is transient and nothing about the codon needs to change. Hankweave supports it in two forms: an automatic policy declared on the codon, and a manual command sent to a live server.

For the automatic form, set onFailure on the codon that needs a failure policy. It accepts abort, retry, or ignore; the default is abort. The schema fields and their defaults:

Before picking a value, match the policy to the failure you want to handle. Each option produces a different server decision:

When you choose retry, the companion settings control how many attempts are made and how long the server waits between them. The defaults are 3 attempts, a 1000 ms base delay, and a 60000 ms maximum delay; their bounds are 1–10, 0–60000 ms, and 0–600000 ms respectively. This example is the shipped retry-configured codon, with the full retryConfig block in place:

      "id": "survey-and-extracts",
      "name": "Prepare native exports and review raw intake",
      "model": "haiku",
      "continuationMode": "fresh",
      "promptFile": "./prompts/survey-and-extracts.md",
      "rigSetup": [
        {
          "type": "copy",
          "copy": {
            "from": "rigs/native-inputs.ts",
            "to": "pipeline/native-inputs.ts"
          }
        },
        {
          "type": "command",
          "command": {
            "run": "bun pipeline/native-inputs.ts"
          }
        }
      ],
      "onFailure": "retry",
      "retryConfig": {
        "maxAttempts": 3,
        "delayMs": 20000,
        "maxDelayMs": 120000
      },

Retry delays grow as delayMs × 2^attempts and are clamped to maxDelayMs. A provider Retry-After hint replaces the computed delay and is clamped to that maximum; a hint above one hour is ignored instead of clamped. Retry counters are in memory and per run. If the server restarts during a retry, the count is lost and the codon remains failed; restore a checkpoint manually before continuing.

Automatic retry only helps if the failure is actually retriable, so check the retriable-versus-permanent failure classification before enabling it. Network failures, timeouts, 429s, and 5xx responses are retriable; authentication, invalid-request, and billing-quota failures are permanent; unrecognized text uses the retriable fallback.

For a manual retry, send codon.redo through the WebSocket control surface. It reruns the most recently attempted codon with the same configuration, whatever its status, and is rejected while a codon is running. With onFailure: "abort", an interactive server stays active for a retriable failure so a client can issue that command. In --headless mode, the same failure fails fast with exit code 1 instead.

The capture below shows an automatic retry running to exhaustion: the policy resolves three times, then the server gives up and shuts down.

[<ts>] [INFO] Resolving failure policy for codon update#0: onFailure=retry, retriable=true
[<ts>] [INFO] Resolving failure policy for codon update#0: onFailure=retry, retriable=true
[<ts>] [INFO] Resolving failure policy for codon update#0: onFailure=retry, retriable=true
[<ts>] [INFO] Codon update#0 exhausted 2 retry attempts, aborting
[<ts>] [DEBUG] [SentinelManager] Shutdown complete
[<ts>] [INFO] State transition: RunFailed
[<ts>] [INFO] Shutdown: Fatal error: Rig setup item 1 (exit code: 1)

Check-it: In the retry capture, find Resolving failure policy for codon update#0: onFailure=retry, retriable=true three times and then the retry-exhaustion message. Before using codon.redo, confirm that the codon is stopped.

Resume without re-running sealed work#

When a run stops for any reason other than a completed hank, you usually do not need to configure anything to continue it. Resume is the default; there is no --resume flag. Re-run the same command to select the newest execution directory for the same data hash, or use -e/--execution <dir> to select a specific directory that has state. Directory selection is covered in execution-directory.

The runtime opens a continuation run after the last completed codon and anchors it at that codon's completion checkpoint, so sealed codons are not run again. A codon in terminal completed state has no valid transitions in that run; --start-new creates a fresh run in which all codons can execute again. The captures below cover two different paths: Resuming from last completed codon: <id> comes from completed-run silent reuse, which does no new codon work; recovery after interruption during write-second (codon 2) takes the thread-failed rollback path and continues after write-line.

Completed-run silent reuse – resume log line:

[<ts>] [INFO] Resuming from last completed codon: write-line in run <id>

Interrupted codon 2 – codon-start counts across the original and resumed runs:

codon.started events across both runs — write-line: 1 · write-second: 2 (codon 1 was sealed before the kill, so it must start exactly once)

Interrupted codon 2 – checkpoint log after recovery:

<sha7> completed:write-second [run:<id>] Write a second line
<sha7> completed:write-line [run:<id>] Write one line
<sha7> Initial checkpoint setup

Check-it: The silent-reuse capture contains Resuming from last completed codon: write-line in run <id> and records zero new codon events. Across the interruption capture, write-line starts once and write-second starts twice; the checkpoint log keeps completed:write-second above completed:write-line and Initial checkpoint setup.

Resume has one behavior that surprises people: a continuation run reuses the persisted execution plan verbatim without re-validating it; only legacy provider-id migration can mutate stored models. The failed codon's resolved model and per-codon budget stay frozen. Run-level ceilings such as maxDollars and maxTimeSeconds are read afresh when a new run starts. Therefore --model and edited-and-accepted hank values do not re-resolve a failed codon on resume.

Use this table when deciding whether an edit belongs in a resumed attempt or a fresh execution. “Saved codon” includes entries already in the persisted plan that have not started yet.

For a fresh isolated attempt at a later stage, explicitly stage the prior artifacts it needs and check them before agent work. Do not assume --start-new imports a previous execution's handoffs.

Two hash checks guard resume against changed inputs. If the hank hash changed, resume warns and asks Continue with modified config?. In a non-interactive or headless environment it prints Non-interactive mode, skipping confirmation prompt. and then fails with Operation cancelled by user. Use -y or -f to acknowledge the changed hank and continue with the saved plan; this confirmation does not re-plan its codons.

If the data hash changed, resume is blocked with an expected-versus-current hash error. The error names three choices: --force to keep the state with the new data, --start-new --force to restart in place, or a different directory. --ignore-data-mismatch is deprecated in favor of --force.

Two terminal cases round out the behavior. If the execution is already RunCompleted, resume starts the server, logs Resuming from last completed codon: <id> and Shutting down server: all codons completed, exits 0, and produces no new codon events. Use --start-new when you want fresh work. If the previous execution thread failed, startup automatically rolls back to the last successful checkpoint before continuing and logs Execution thread failed, rolling back....

Check-it: For a changed hank, verify the two non-interactive messages. For a completed execution, verify the all-codons-completed message and a zero event-count delta. After a thread failure, look for the rollback message before the continuation run proceeds.

Roll back when file state is wrong#

Retry and resume assume the files on disk are still a good basis for the next attempt. When they are not, roll back. Rollback is the right move when file state is corrupted or when you want a different approach from a known-good state. It is a WebSocket/TUI operation, not a CLI action; 0.10.0 has no CLI rollback flag. Use checkpoint.list to enumerate targets, then choose rollback.toLastSuccess, rollback.toCodon, or rollback.toCheckpoint. The rollback.toCodon checkpoint type is one of start, end, rig-setup, completed, error, or skipped; rollback.toCheckpoint takes a commit SHA. The command catalog slice below shows the exact shapes the server accepts:

    type: z.literal("codon.next"),
  }),
  z.object({
    id: z.string(),
    type: z.literal("codon.skip"),
  }),
  z.object({
    id: z.string(),
    type: z.literal("codon.redo"),
  }),
  z.object({
    id: z.string(),
    type: z.literal("server.shutdown"),
    data: z
      .object({
        reason: z.string().optional(),
      })
      .optional(),
  }),
  z.object({
    id: z.string(),
    type: z.literal("server.force_shutdown"),
    data: z
      .object({
        reason: z.string().optional(),
      })
      .optional(),
  }),

  // Query checkpoints
  z.object({
    id: z.string(),
    type: z.literal("checkpoint.list"),
    data: z
      .object({
        runId: z.string().optional(), // Defaults to current run
      })
      .optional(),
  }),

  // Force stop current codon
  z.object({
    id: z.string(),
    type: z.literal("codon.forceStop"),
    data: z
      .object({
        reason: z.string().optional(),
      })
      .optional(),
  }),

  // Rollback to specific checkpoint
  z.object({
    id: z.string(),
    type: z.literal("rollback.toCheckpoint"),
    data: z.object({
      checkpointSha: z.string(),
      autoRestart: z.boolean().optional().default(false),
    }),
  }),

  // Rollback to codon + checkpoint type
  z.object({
    id: z.string(),
    type: z.literal("rollback.toCodon"),
    data: z.object({
      codonId: codonIdSchema,
      checkpointType: z.enum(["start", "end", "rig-setup", "completed", "error", "skipped"]),
      autoRestart: z.boolean().optional().default(false),
    }),
  }),

  // Rollback to last successful codon
  z.object({
    id: z.string(),
    type: z.literal("rollback.toLastSuccess"),
    data: z
      .object({
        autoRestart: z.boolean().optional().default(false),
      })
      .optional(),
  }),

  // Ping commands for testing
  z.object({
    id: z.string(),
    type: z.literal("ping"),
  }),
  z.object({

All three rollback commands default autoRestart to false. With that default, rollback ends at server.idle with reason rollback-completed and message Rollback completed. Use 'codon.next' to continue. Use codon.next, codon.skip, codon.forceStop, or codon.redo for manual sequencing around the rollback. The command signatures and WebSocket handshake belong to the protocol reference.

While rollback is in progress, state-modifying WebSocket commands are blocked. checkpoint.list, history.sync, shutdown commands, and pings remain allowed. Wait for rollback to finish before sending another state-changing command.

One failure mode needs special handling. If resume-after-failure reports Target checkpoint a2f1b13… not found in manifest, the failed codon's workspace files may be deleted before the process dies; the archive manifest then conservatively restores all entries. The failed codon's files survive in its error: checkpoint in the shadow Git repository. Inspect that checkpoint using the explicit repository paths described in checkpoints, not the project's .git directory:

cd <execution-dir>
export GIT_DIR=.hankweave/checkpoints/.hankweavecheckpoints
export GIT_WORK_TREE=agentRoot
git --git-dir="$GIT_DIR" log --oneline --all

Do not confuse this operator rollback with the automatic rollback that resume performs after a thread failure. checkpointSha in an automatic resume rollback identifies the checkpoint commit; it is not an operator rollback.toLastSuccess command. When a later codon was interrupted, the continuation's startingConditions records reason: "rollback" and the afterCodon anchor. If codon 1 is killed before any successful checkpoint exists, the logs say Did not find any successful codon to rollback to and No checkpoints found in execution history; codon 1 then runs from scratch. The state and log captures below show both situations:

{
  "currentRunId": null,
  "latestRun": {
    "runId": "<id>",
    "status": "completed",
    "startingConditions": {
      "type": "continuation",
      "source": {
        "runId": "<id>",
        "afterCodon": "write-line",
        "checkpointSha": "<sha>"
      },
      "reason": "rollback"
    },
    "codons": [
      {
        "codonId": "write-second",
        "status": "completed",
        "exitCode": 0
      }
    ]
  },
  "runs": [
    {
      "runId": "<id>",
      "status": "completed",
      "startingConditions": {
        "type": "continuation",
        "source": {
          "runId": "<id>",
          "afterCodon": "write-line",
          "checkpointSha": "<sha>"
        },
        "reason": "rollback"
      },
      "codons": [
        {
          "codonId": "write-second",
          "status": "completed",
          "exitCode": 0
        }
      ]
    },
    {
      "runId": "<id>",
      "status": "completed",
      "startingConditions": {
        "type": "fresh"
      },
      "codons": [
        {
          "codonId": "write-line",
          "status": "completed",
          "exitCode": 0
        },
        {
          "codonId": "write-second",
          "status": "failed",
          "exitCode": -1
        }
      ]
    }
  ]
}

[<ts>] [INFO] [DEBUG] Initializing checkpoints...
[<ts>] [INFO] Fresh run <id> - branch will be created on first checkpoint
[<ts>] [DEBUG] Built execution thread: 0 codons across 1 runs, next codon: write-line
[<ts>] [INFO] Added checkpoint patterns: out.txt
[<ts>] [INFO] Shutting down server: SIGTERM
[<ts>] [INFO] [DEBUG] Initializing checkpoints...
[<ts>] [DEBUG] Built execution thread: 1 codons across 1 runs, next codon: write-second
[<ts>] [DEBUG] Built execution thread: 1 codons across 1 runs, next codon: write-second
[<ts>] [INFO] Did not find any successful codon to rollback to. Going to look for a checkpoint in the thread.
[<ts>] [ERROR] No checkpoints found in execution history
[<ts>] [INFO] Fresh run <id> - branch will be created on first checkpoint
[<ts>] [DEBUG] Built execution thread: 0 codons across 1 runs, next codon: write-line
[<ts>] [INFO] Added checkpoint patterns: out.txt
[<ts>] [INFO] [CHECKPOINT-DEBUG] Creating checkpoint for codon write-line with status completed
[<ts>] [INFO] [CHECKPOINT-DEBUG] Created checkpoint: <sha> (completed) on branch run-<id>
[<ts>] [DEBUG] Built execution thread: 1 codons across 1 runs, next codon: write-second
[<ts>] [DEBUG] Built execution thread: 1 codons across 1 runs, next codon: write-second
[<ts>] [INFO] Added checkpoint patterns: out.txt
[<ts>] [INFO] Added checkpoint patterns: second.txt
[<ts>] [INFO] [CHECKPOINT-DEBUG] Creating checkpoint for codon write-second with status completed
[<ts>] [INFO] [CHECKPOINT-DEBUG] Created checkpoint: <sha> (completed) on branch run-<id>
[<ts>] [DEBUG] Built execution thread: 2 codons across 1 runs, next codon: none
[<ts>] [DEBUG] Built execution thread: 2 codons across 1 runs, next codon: none
[<ts>] [INFO] Shutting down server: all codons completed
[<ts>] [INFO] Shutting down sentinel manager...
[<ts>] [INFO] Shutdown: all codons completed (exit code: 0)

Check-it: Confirm reason: "rollback" in the state capture. In the no-successful-checkpoint capture, verify both quoted messages. If the manifest error occurs, inspect the error: checkpoint before sending another state-changing command.

Recover after an exhaust extension fails#

An exhaustWithPrompt extension is attempted only after a clean codon exit: exit code 0, a received result, no failure reason, no interruption, intact budget, and fewer than maxExtensions extensions. If the extension attempt fails, it enters the normal failure policy even though the codon already sealed a complete checkpoint; there is no post-seal exception. This failure mode has been observed with a provider 429 quota failure and an idle timeout, and a subsequent resume can select the sealed codon again.

maxExtensions is the extension cap; its documented default is 100. A successful extension emits the journaled, sentinel (event observer)-routed codon.extended event with the extension number and cumulative token and cost totals:

Check-it: Inspect codon.extended for extensionNumber, cumulativeTokens, and cumulativeCost. If an extension fails after sealing, apply the normal failure policy and use the checkpoint history to choose the next recovery move.

Replay recorded codon output#

Sometimes you do not want to recover a run at all; you want to see what it did. Use --replay <dir> to reproduce recorded codon output from per-codon logs on a temporary copy. It is a recorded-output reproduction, not a guarantee that every provider startup is avoided or that a replay is universally zero-cost.

Replay the kept execution with the pinned launcher:

bunx hankweave@0.10.0 --replay <kept-execution-dir>

Replay discovers the hank and data paths from <dir>/.hankweave/execution-meta.json, copies the execution directory to a temporary directory, removes the copied runtime lock, and preserves the original. It starts a fresh replay run even when the source execution is complete. It requires every codon's log file; if one is missing, it hard-errors and lists the missing file instead of falling back to a real execution. Rig setup is skipped because the copied directory already holds the post-rig-setup filesystem state; sentinels are skipped because replay reproduces only recorded codon output–sentinel analysis would make fresh provider calls, and that behavior was not recorded in the codon logs. The temporary copy is removed when the replay process exits.

Two details matter when you script around replay. --replay is parsed by the binary but omitted from --help. Replay pacing comes from recorded timestamps; for recordings without usable timestamps, HANKWEAVE_REPLAY_SPEED_MS sets the per-line floor, which defaults to 5 ms. --replay and --execution are mutually exclusive, and the exit-code-1 error names both options.

The capture below shows a successful replay of a completed execution, including the metadata lookups and the temporary copy:

╭────────────────────────────────────────────────────────────────────╮
│  Hankweave v0.10.0                                                 │
│  darwin arm64 • node v23.8.0                                       │
╰────────────────────────────────────────────────────────────────────╯

[REPLAY] Using hank config from execution metadata: <workspace>/fixtures/scenarios/silent-reuse/hank.json
[REPLAY] Using data source from execution metadata: <workspace>/fixtures/scenarios/silent-reuse/data
[REPLAY] Copied execution dir to <capture-attempt><tmp>/hankweave-replay-1788702367070-4s0v
Resuming: hankweave-replay-1788702367070-4s0v
  Source → data
  Exec   → <capture-attempt><tmp>/hankweave-replay-1788702367070-4s0v
  SDKs   → Claude node_modules ✓

╭──────────────────────────────────────────────────────────────────────────────╮
│  Silent reuse v1.0.0                                                         │
│  1 codon • 0 loops                                                           │
╰──────────────────────────────────────────────────────────────────────────────╯

└─ [1] write-line (Write one line)
      model: haiku │ mode: fresh │ prompts: 1 (2 lines)
      checkpointedGlobs: 1


══════════════════════════════════════════════════
  Hankweave Server Started
  WebSocket: ws://localhost:<port>
══════════════════════════════════════════════════

Running in headless mode on port <port>
➜ Listening on: http://localhost:<port>/ (all interfaces)

exit=0

Check-it: The completed-execution capture shows both metadata lookups, the copied replay directory, Resuming: hankweave-replay-…, and exit 0. Its manifest also checks that the original execution state and journal remain byte-identical. Do not use this capture to claim that changed rigs, external side effects, or every provider call were replayed.

Choose retry, edit, rollback, or a new run#

With the failure record in hand, the four moves reduce to one question: what needs to change before the next attempt? The table maps each answer to its move:

↗ Enlarge

                      +------------------------+
                      | what needs to change?  |
                      +-----------+------------+
        +--------------+----------+-----------+-----------------------+
        | a retriable  | the prompt or        | corrupted | the model or the
        | failure      | instructions         | file state| persisted plan
        v              v                      v           v
+-----------------+ +---------------------+ +------------------+ +---------------------+
| retry the codon | | edit the prompt file| | roll back to a   | | start over with     |
|                 | | and resume          | | checkpoint       | | --start-new         |
+-----------------+ +---------------------+ +------------------+ +---------------------+

Rollback, like plain resume, starts a continuation that reuses the persisted execution plan; it does not re-resolve edited model or budget fields. Use --start-new for an independent model-tier comparison, with a separate execution and output directory. Do not present a chapter directory as a Git tag or a checkpoint name; run the next chapter from its shipped directory with its own hank.

There are no --start-at or --end-at codon-range flags. Isolate one codon with the state file and a targeted rollback instead.

Check-it: Name the deciding signal before acting: failureReason.retriable, the prompt-file change, corrupted file state, or the need for --start-new.

Avoid recovery patterns that compound failures#

The moves above are safe in themselves; the damage usually comes from applying the wrong one repeatedly. Four patterns account for most of it:

1. Do not retry an unchanged predictable failure. Check failureReason.retriable first. A wrong prompt or missing file will reproduce instead of benefiting from another attempt.
2. Rewrite instead of patching a prompt forever. Appending a fix after each failure accumulates contradictory instructions. Identify what the agent misunderstood, then rewrite the prompt with a clear structure.
3. Isolate instead of deleting codons. Removing codons can remove the codon that exposes the bug rather than the codon that causes it. Start with the state file and a targeted rollback; do not use the nonexistent --start-at or --end-at flags.
4. Keep useful partial progress. Roll back to the last successful checkpoint, not the beginning, so valid work completed before the failure remains available.

Check-it: Record the deciding signal, the rewritten prompt when instructions were wrong, or the failed codon and rollback target when file state was involved.