Troubleshoot a run by its exact symptom

When a hank run fails, the fastest path to a fix is the literal text it printed. This page is a symptom lookup: you bring the exact error string, and the tables below match it to the phase of the run where it occurred, the cause, and the recovery step. Each row also points to the page that owns the underlying topic, so you can go deeper once the immediate failure is cleared.

The page works through the run lifecycle in order: failures that stop a run before it starts, failures that kill it mid-flight, and messages that look like failures but are side effects of something else. A final section covers what to gather when no row matches.

Match your error text to a phase#

Start with the exact error string from your hank, the configuration being run. We'll match the text to a lifecycle phase and choose the next action. A codon is one sealed agent task in the sequence; its failure text is the lookup key.

1. Copy the string from console or headless (non-interactive) stderr, .hankweave/logs/server.log, the failed codon's failureReason in .hankweave/state.json, or the agent log at claudeLogPath.
2. Mark the phase: before-run, preflight, running, or completed. Startup self-test rows apply both with --validate and when a run starts.
3. Match the literal text below instead of paraphrasing it. Several classes share keywords, and the exact wording is what separates, for example, a retriable 429 from a quota 429.
4. Follow the owning page when the fix needs recovery, model selection, failure classes, or artifact inspection.

If you don't yet have the error string, start with the Runbook for the lifecycle or Observe and debug for the four-stop diagnosis order.

Fix failures before the run starts#

These symptoms occur before a codon runs or during the checks before it runs: the startup self-test, the model catalog preflight, resume confirmation, and execution-directory selection. Because nothing has executed yet, the fix is usually a credential, a model spelling, or a command-line flag rather than a change to the hank itself.

[<ts>] [ERROR] Self-test FAILED: Some checks failed
[<ts>] [ERROR]   - authentication: ✗ No authentication found (set ANTHROPIC_API_KEY)
[ERROR] Server startup failed!
Error message: Self-test failed for 1 model(s):
  - Claude Haiku 4.5 (latest) (anthropic/claude-haiku-4-5): Some checks failed
…

Two rows in that table need their current scope spelled out. The missing-key self-test failure is classified check, not auth: the auth class requires a failed check literally named api_key, which no shipped harness emits, while the message No authentication found (set ANTHROPIC_API_KEY) is the failed authentication check's own text. The full self-test class list belongs to errors and exit codes.

The hank-changed check also applies only when you resume an explicit -e/--execution directory whose recorded hank hash differs. A managed resume (no -e) never reads the hank hash, so it prints no warning, prompt, or refusal in any mode. On the explicit path, a non-interactive run (headless, CI, piped stdio) fails closed before the warning with a self-contained error naming add -y, --start-new and --start-new --force; interactively it prints Continue with modified config? and stops with Operation cancelled by user. on a no. -y/--yes (or --force) is what lets that explicit resume proceed, and --start-new runs fresh. Resume semantics are owned by resume, rollback, and retry.

The first row is the most common starting point, so its expected failure output is captured alongside the table. After setting ANTHROPIC_API_KEY, validate again and look for the captured GOOD TO RUN! box, which confirms the self-test now passes:

╭─ GOOD TO RUN! ─────────────────────────────────────────────────────╮
│  1 codons • 1 prompts • 0 system prompts • 0 rigs • 1 checkpoints  │
╰────────────────────────────────────────────────────────────────────╯

Recover when the run dies mid-flight#

Once codons are running, failures come from the provider, the budget caps, rig execution (a rig is a setup step around a codon), or the container environment. Use the running-phase rows below. When an error could match more than one class, the classification rules are tried in matchOrder order and the lowest matchOrder wins; the reference rows are included first so you can see why, for example, a usage-limit 429 classifies as billing-quota rather than bare-429.

Pi gives a busy turn or tool call at least 300 seconds of silence: its busy threshold is max(shimIdleTimeout, 300s). The 120-second fallback above is the idle threshold, not a universal maximum tool duration. An unexplained SIGKILL or SIGTERM alone does not identify an idle timeout; correlate it with the watchdog, budget, cancellation, and host logs.

[<ts>] [ERROR] Query execution failed for codon summarize-notes
Session ID: N/A
Working directory: /executions/<exec-id>
Error type: Error
Error message: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
Stack trace:
Error: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
    at ak.getProcessExitError (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:121:11456)
    at ChildProcess.n (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:121:15696)
    at Object.onceWrapper (node:events:634:26)
    at ChildProcess.emit (node:events:531:35)
    at m (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:121:4918)
    at ChildProcess.<anonymous> (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:121:5122)
    at Object.onceWrapper (node:events:634:26)
    at ChildProcess.emit (node:events:531:35)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)
[<ts>] [ERROR] Query error: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
[<ts>] [ERROR] [CodonRunner] SDK crash classified permanent — routing to exit path for failure policy: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons

exit=1

Two rows in that table deserve expansion. For a budget cap, the runtime requests the harness stop (SIGTERM) when a cap trips, and the codon's onExceeded policy then decides the outcome: the default "complete" finishes the codon early with a budget limit reached info event, while "fail" fails it non-retriably and stores a failureReason beginning Budget exceeded: …. The token fields are maxOutputTokens and maxContextTokens in BudgetLimits.

For the quota and model rows, the persisted plan is the constraint to remember: a continuation (resume or rollback) reuses the plan, so every entry already in it, failed or not yet started, runs on its persisted model with its persisted per-codon budget cap. --model re-resolves nothing on a resume; changing a model, or an edited per-codon cap, requires --start-new to build a fresh plan. Recovery procedures are owned by resume, rollback, and retry.

On the health-check row's proxy note: the proxy is off by default, so --proxy turns it on and --without-proxy merely restates the default. The flag is listed in --help, but it is redundant rather than a primary interface.

The rig row deserves a closer look, because its fix is a pattern rather than a one-liner. Command rigs execute in the execution workspace, so a script that lives in the hank directory must be copied into the workspace before it can run. The anchor-hank fixture's normalize-aster rigSetup does exactly this – create a directory, copy the script, then execute the copied path:

      "rigSetup": [
        {
          "type": "command",
          "command": {
            "run": "mkdir -p pipeline"
          }
        },
        {
          "type": "copy",
          "copy": {
            "from": "rigs/preflight.ts",
            "to": "pipeline/preflight.ts"
          }
        },
        {
          "type": "command",
          "command": {
            "run": "bun pipeline/preflight.ts"
          }
        }
      ],

After a transport or quota recovery and resume, confirm the run actually finished: the journal should show codon.completed followed by RunCompleted, as in this capture. A finished execution may be dieted, so read the journal while the run is live, or restore it with --restore-journal before opening it offline:

{"id": "<id>", "timestamp": "<ts>", "type": "codon.completed", "data": {"codonId": "summarize-notes", "success": true, "cost":"<n>", "duration":"<n>", "exitStatus": {"type": "success"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunCompleted", "runId": "<id>", "transition": {"type": "RunCompleted", "data": {"runId": "<id>"}}, "resultingState": {"currentRunId": null, "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}

Separate misleading messages from real failures#

Some messages describe a side effect or an earlier cause rather than the failure itself. A budget trip surfaces as a user abort; a telemetry lookup failure prints ERROR on a run that succeeded. Match neighboring log lines before choosing a recovery path.

The completed-reuse row is the one to read carefully, because the reuse is announced, not silent. Without --start-new, the runtime resumes the newest execution whose data signature matches and prints Resuming execution in: <path>; when that execution is already RunCompleted, it starts an empty continuation and announces Resumed completed execution <id> — nothing to do. All codons already completed. Use --start-new to run fresh. (on stdout right after ➜ Listening on under --headless; as an Info: line in the TUI). It runs no codon, the journal gains no codon events, and the process exits 0 because the run completed.

After the --start-new relaunch, look for the captured new-directory line and the fresh startup block – their presence is what distinguishes a real fresh run from a resumed completed execution:

Created new execution directory: ~/.hankweave-executions/<exec-id>
New execution: <exec-id>
  Source → data
  Exec   → ~/.hankweave-executions/<exec-id>
  SDKs   → Claude node_modules ✓

╭──────────────────────────────────────────────────────────────────────────────╮
│  Minimal single provider v1.0.0                                              │
  …

Escalate symptoms this table does not cover#

If the message is too vague to diagnose, we'll gather the evidence in this order:

1. Inspect state.json.
2. Read the exact claudeLogPath stored there.
3. Read the journal.
4. Diff the checkpoint – a run-boundary marker.

For failure classes, failureReason, and exit codes, use errors and exit codes. For retry, resume, rollback, and replay, use resume, rollback, and retry. Use observe and debug for the artifact sequence and help for the great-bug-report recipe.

If an authentication or model error survives editing hank.json, check whether it comes from the wizard, a template, or the pinned health-check model. The separate model-selection paths are explained in authentication and models.