Troubleshoot a run by its exact symptom

When a hank run fails, the fastest path to a fix is the literal text it printed. This page is a symptom lookup: you bring the exact error string, and the tables below match it to the phase of the run where it occurred, the cause, and the recovery step. Each row also points to the page that owns the underlying topic, so you can go deeper once the immediate failure is cleared.

The page works through the run lifecycle in order: failures that stop a run before it starts, failures that kill it mid-flight, and messages that look like failures but are side effects of something else. A final section covers what to gather when no row matches.

Match your error text to a phase#

Start with the exact error string from your hank, the configuration being run. We'll match the text to a lifecycle phase and choose the next action. A codon is one sealed agent task in the sequence; its failure text is the lookup key.

1. Copy the string from console or headless (non-interactive) stderr, .hankweave/logs/server.log, the failed codon's failureReason in .hankweave/state.json, or the agent log at claudeLogPath.
2. Mark the phase: before-run, preflight, running, or completed. Startup self-test rows apply both with --validate and when a run starts.
3. Match the literal text below instead of paraphrasing it. Several classes share keywords, and the exact wording is what separates, for example, a retriable 429 from a quota 429.
4. Follow the owning page when the fix needs recovery, model selection, failure classes, or artifact inspection.

If you don't yet have the error string, start with the Runbook for the lifecycle or Observe and debug for the four-stop diagnosis order.

Fix failures before the run starts#

These symptoms occur before a codon runs or during the checks before it runs: the startup self-test, the model catalog preflight, resume confirmation, and execution-directory selection. Because nothing has executed yet, the fix is usually a credential, a model spelling, or a command-line flag rather than a change to the hank itself.

[<ts>] [ERROR] Self-test FAILED: Some checks failed
[<ts>] [ERROR]   - authentication: ✗ No authentication found (set ANTHROPIC_API_KEY)
[ERROR] Server startup failed!
Error message: Self-test failed for 1 model(s):
  - Claude Haiku 4.5 (latest) (anthropic/claude-haiku-4-5): Some checks failed
…

The first row is the most common starting point, so its expected failure output is captured alongside the table. After setting ANTHROPIC_API_KEY, validate again and look for the captured GOOD TO RUN! box, which confirms the self-test now passes:

╭─ GOOD TO RUN! ─────────────────────────────────────────────────────╮
│  1 codons • 1 prompts • 0 system prompts • 0 rigs • 1 checkpoints  │
╰────────────────────────────────────────────────────────────────────╯

Recover when the run dies mid-flight#

Once codons are running, failures come from the provider, the budget caps, rig execution, or the container environment. Use the running-phase rows below. When an error could match more than one class, the classification rules are tried in matchOrder order and the lowest matchOrder wins; the reference rows are included first so you can see why, for example, a usage-limit 429 classifies as billing-quota rather than bare-429.

Pi gives a busy turn or tool call at least 300 seconds of silence: its busy threshold is max(shimIdleTimeout, 300s). The 120-second fallback above is the idle threshold, not a universal maximum tool duration. An unexplained SIGKILL or SIGTERM alone does not identify an idle timeout; correlate it with the watchdog, budget, cancellation, and host logs.

[<ts>] [ERROR] Query execution failed for codon summarize-notes
Session ID: N/A
Working directory: /executions/<exec-id>
Error type: Error
Error message: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
Stack trace:
Error: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
    at Rk.getProcessExitError (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:118:11055)
    at ChildProcess.n (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:118:15335)
    at Object.onceWrapper (node:events:634:26)
    at ChildProcess.emit (node:events:531:35)
    at m (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:118:4681)
    at ChildProcess.<anonymous> (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:118:4885)
    at Object.onceWrapper (node:events:634:26)
    at ChildProcess.emit (node:events:531:35)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)
[<ts>] [ERROR] Query error: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
[<ts>] [ERROR] [CodonRunner] SDK crash classified permanent — routing to exit path for failure policy: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons

exit=1

The rig row deserves a closer look, because its fix is a pattern rather than a one-liner. Command rigs execute in the execution workspace, so a script that lives in the hank directory must be copied into the workspace before it can run. The anchor-hank fixture's normalize-aster rigSetup does exactly this – create a directory, copy the script, then execute the copied path:

      "rigSetup": [
        {
          "type": "command",
          "command": {
            "run": "mkdir -p pipeline"
          }
        },
        {
          "type": "copy",
          "copy": {
            "from": "rigs/preflight.ts",
            "to": "pipeline/preflight.ts"
          }
        },
        {
          "type": "command",
          "command": {
            "run": "bun pipeline/preflight.ts"
          }
        }
      ],

For a budget cap, onExceeded: "complete" finishes the codon early with a budget limit reached info event. onExceeded: "fail" fails it without retry and stores a failureReason beginning Budget exceeded: …. The token fields are maxOutputTokens and maxContextTokens in BudgetLimits.

After a transport or quota recovery and resume, confirm the run actually finished: the journal should show codon.completed followed by RunCompleted, as in this capture:

{"id": "<id>", "timestamp": "<ts>", "type": "codon.completed", "data": {"codonId": "summarize-notes", "success": true, "cost":"<n>", "duration":"<n>", "exitStatus": {"type": "success"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunCompleted", "runId": "<id>", "transition": {"type": "RunCompleted", "data": {"runId": "<id>"}}, "resultingState": {"currentRunId": null, "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}

Separate misleading messages from real failures#

Some messages describe a side effect or an earlier cause rather than the failure itself. A budget trip surfaces as a user abort; a telemetry lookup failure prints ERROR on a run that succeeded. Match neighboring log lines before choosing a recovery path.

After the --start-new relaunch, look for the captured new-directory line and the fresh startup block – their presence is what distinguishes a real fresh run from a silent reuse:

Created new execution directory: ~/.hankweave-executions/<exec-id>
New execution: <exec-id>
  Source → data
  Exec   → ~/.hankweave-executions/<exec-id>
  SDKs   → Claude node_modules ✓

╭──────────────────────────────────────────────────────────────────────────────╮
│  Minimal single provider v1.0.0                                              │
  …

Escalate symptoms this table does not cover#

If the message is too vague to diagnose, we'll gather the evidence in this order:

1. Inspect state.json.
2. Read the exact claudeLogPath stored there.
3. Read the journal.
4. Diff the checkpoint – a run-boundary marker.

For failure classes, failureReason, and exit codes, use errors and exit codes. For retry, resume, rollback, and replay, use resume, rollback, and retry. Use observe and debug for the artifact sequence and help for the great-bug-report recipe.

If an authentication or model error survives editing hank.json, check whether it comes from the wizard, a template, or the pinned health-check model. The separate model-selection paths are explained in authentication and models.