# Troubleshoot a run by its exact symptom

When a hank run fails, the fastest path to a fix is the literal text it printed. This page is a symptom lookup: you bring the exact error string, and the tables below match it to the phase of the run where it occurred, the cause, and the recovery step. Each row also points to the page that owns the underlying topic, so you can go deeper once the immediate failure is cleared.

The page works through the run lifecycle in order: failures that stop a run before it starts, failures that kill it mid-flight, and messages that look like failures but are side effects of something else. A final section covers what to gather when no row matches.

## Match your error text to a phase

Start with the exact error string from your hank, the configuration being run. We'll match the text to a lifecycle phase and choose the next action. A codon is one sealed agent task in the sequence; its failure text is the lookup key.

1. Copy the string from console or headless (non-interactive) stderr, `.hankweave/logs/server.log`, the failed codon's `failureReason` in `.hankweave/state.json`, or the agent log at `claudeLogPath`.
2. Mark the phase: `before-run`, `preflight`, `running`, or `completed`. Startup self-test rows apply both with `--validate` and when a run starts.
3. Match the literal text below instead of paraphrasing it. Several classes share keywords, and the exact wording is what separates, for example, a retriable `429` from a quota `429`.
4. Follow the owning page when the fix needs recovery, model selection, failure classes, or artifact inspection.

If you don't yet have the error string, start with the [Runbook](/operate/runbook) for the lifecycle or [Observe and debug](/operate/observe-and-debug) for the four-stop diagnosis order.

## Fix failures before the run starts

These symptoms occur before a codon runs or during the checks before it runs: the startup self-test, the model catalog preflight, resume confirmation, and execution-directory selection. Because nothing has executed yet, the fix is usually a credential, a model spelling, or a command-line flag rather than a change to the hank itself.

| Symptom                                                                                                                                                                                           | Phase                           | Cause                                                                                                                                                                                                                                                  | Fix                                                                                                                                                                                                                                                                                                  | See                                                                                                                         |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `Self-test FAILED: Some checks failed`<br />`- authentication: ✗ No authentication found (set ANTHROPIC_API_KEY)`<br />`[ERROR] Server startup failed!`<br />`Self-test failed for 1 model(s): …` | Before-run: startup self-test   | Every unique codon model receives a harness self-test before any codon runs. One failure stops execution. Since 0.7.0, startup self-test failures carry a failure class; this row is `auth`.                                                           | For the default direct-Anthropic route, export `ANTHROPIC_API_KEY`, then validate again. The check names the variable, but passing the presence check is not proof that the credential is valid. The full self-test class list belongs to [errors and exit codes](/reference/errors-and-exit-codes). | [Authentication and models](/operate/authentication-and-models) · [Environment variables](/reference/environment-variables) |
| `Pi model not found: <provider>/<model>. Available '<provider>' models: …`<br />`Unknown pi provider '<provider>'. Known providers: …`                                                            | Preflight: model catalog check  | The model catalog–the list of models Pi can serve–cannot serve the spelling, so the run fails before any codon starts.                                                                                                                                 | Re-spell the model as an id the catalog can serve. The `glm-5.3`/`zai-coding-plan` case and decision table are in [authentication and models](/operate/authentication-and-models) and [model resolution](/reference/model-resolution).                                                               | [Authentication and models](/operate/authentication-and-models) · [Model resolution](/reference/model-resolution)           |
| `WARNING: hank.json has changed since last execution.`<br />`Continue with modified config?`<br />`Operation cancelled by user.`                                                                  | Before-run: resume confirmation | The hank hash changed; the warning shows previous and current hash prefixes. In non-interactive mode, confirmation defaults to cancel.                                                                                                                 | `-y` acknowledges the changed file but does not rebuild the saved codons. Rig commands, model, per-codon budget, output definitions, and prompt paths remain frozen; contents at saved prompt-file paths are reread. Use the change/resume table before choosing `--start-new`.                      | [Resume change table](/operate/resume-rollback-and-retry#resume-without-re-running-sealed-work)                             |
| `Data source not found: <path>`                                                                                                                                                                   | Before-run or preflight         | Run setup cannot find the data source. `--validate` reports the same setup failure.                                                                                                                                                                    | Run from the data directory or pass the data path positionally.                                                                                                                                                                                                                                      | [Runbook](/operate/runbook)                                                                                                 |
| `Cannot create execution inside another execution directory`<br />`Cannot create new execution in ~/.hankweave-executions/.`                                                                      | Before-run: execution selection | These checks concern the explicit `--execution` path: an existing path containing both `/.hankweave-executions/` and `/data` hits the nested-execution guard, while a new `-e` target under the managed root without `execution-meta.json` is refused. | Correct the `--execution` argument: choose a different explicit execution directory, or omit `-e` and let the runtime allocate one. Keep the input source and execution directory distinct; changing shell directory alone does not fix an invalid `-e` target.                                      | [Runbook](/operate/runbook) · [Execution directory](/reference/execution-directory)                                         |

```text
[<ts>] [ERROR] Self-test FAILED: Some checks failed
[<ts>] [ERROR]   - authentication: ✗ No authentication found (set ANTHROPIC_API_KEY)
[ERROR] Server startup failed!
Error message: Self-test failed for 1 model(s):
  - Claude Haiku 4.5 (latest) (anthropic/claude-haiku-4-5): Some checks failed
…
```

The first row is the most common starting point, so its expected failure output is captured alongside the table. After setting `ANTHROPIC_API_KEY`, validate again and look for the captured `GOOD TO RUN!` box, which confirms the self-test now passes:

```text
╭─ GOOD TO RUN! ─────────────────────────────────────────────────────╮
│  1 codons • 1 prompts • 0 system prompts • 0 rigs • 1 checkpoints  │
╰────────────────────────────────────────────────────────────────────╯
```

> **VersionNote:** Since 0.10.0, the Pi catalog preflight moves “model not found” and “unknown provider” failures to config load; older versions surfaced the miss when the codon started.

## Recover when the run dies mid-flight

Once codons are running, failures come from the provider, the budget caps, rig execution, or the container environment. Use the running-phase rows below. When an error could match more than one class, the classification rules are tried in `matchOrder` order and the lowest `matchOrder` wins; the reference rows are included first so you can see why, for example, a usage-limit `429` classifies as `billing-quota` rather than `bare-429`.

| id            | pattern                                                                                                                                                                                                                                       | classification | retriable | matchOrder | receipts                                                         |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | --------- | ---------- | ---------------------------------------------------------------- |
| auth          | \b401\b OR \b403\b OR authentication OR unauthorized OR forbidden OR api key OR oauth                                                                                                                                                         | api-error      | false     | 14         | error-classification.ts:373-387, error-classification.ts:291-293 |
| bare-429      | \b429\b                                                                                                                                                                                                                                       | rate-limit     | true      | 16         | error-classification.ts:414-423                                  |
| billing-quota | credit \| billing \| insufficient \| quota \| available balance \| out of budget \| /usage\s?limit/                                                                                                                                           | api-error      | false     | 2          | error-classification.ts:322-337                                  |
| timeout       | timeout OR timed out                                                                                                                                                                                                                          | timeout        | true      | 13         | error-classification.ts:363-370                                  |
| transport     | socket OR econnreset OR econnrefused OR econnaborted OR epipe OR etimedout OR enotfound OR eai\_again OR unable to connect OR fetch failed OR connection reset OR connection closed OR connection error OR (stream AND terminated) OR network | api-error      | true      | 17         | error-classification.ts:425-452                                  |

| Symptom                                                                                                                                                                                     | Phase                                | Cause                                                                                                                                                                                                                                                                                                                                                                                                                | Fix                                                                                                                                                                                                                                                                                                                                                              | See                                                                                                                                                                           |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A provider `429` whose text matches the usage-limit class; an observed case was a zai five-hour-window `429` with code `1308`.                                                              | Running: provider quota              | Since 0.8.0, `billing-quota` wins before `bare-429`, so this case is non-retriable and the run exits `1`. Provider quota windows are not modeled by the runtime's `maxDollars` or `maxTimeSeconds` caps.                                                                                                                                                                                                             | Stagger codons using the same provider, split providers across a chain, and use `--model` to target another provider in a fresh run. A resumed failed codon's saved model choice is not re-resolved by `--model`; use `--start-new` when that model must change. Rollback also reuses the persisted plan.                                                        | [Resume, rollback, and retry](/operate/resume-rollback-and-retry) · [Errors and exit codes](/reference/errors-and-exit-codes)                                                 |
| `Connection error`, `fetch failed`, `ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, or `ENOTFOUND`                                                                                               | Running or preflight: transport      | These strings match the retriable transport class. A persistent startup failure can indicate DNS, VPN, or proxy configuration.                                                                                                                                                                                                                                                                                       | Let a codon retry under `onFailure: "retry"`; retries use exponential backoff and honor the provider's `Retry-After`. If the failure is in startup, inspect name lookup, VPN, and proxy settings.                                                                                                                                                                | [Authentication and models](/operate/authentication-and-models) · [Codons](/concepts/codons)                                                                                  |
| `Idle timeout: no events received for <N>ms`                                                                                                                                                | Running: harness inactivity watchdog | No qualifying harness event arrived within its effective silence limit. The Claude Agent SDK fallback is 180 seconds; embedded Pi's idle fallback is 120 seconds.                                                                                                                                                                                                                                                    | Inspect the last agent/tool event and transport logs. A codon's `shimIdleTimeout` overrides the resolved runtime/hank default, which overrides the harness fallback; `--shim-idle-timeout` sets the runtime default. Choose an override for the expected silent work, not as a substitute for diagnosing a hung request. This is distinct from `maxTimeSeconds`. | [Timeout fields](/reference/hank-json#control-failure-and-resource-limits) · [CLI](/reference/cli)                                                                            |
| `Health check timeout`                                                                                                                                                                      | Preflight: provider health probe     | This matches the retriable timeout class. The provider probe is a real `generateText` call with a 5-second abort per candidate model; when candidates are exhausted, the provider is marked `healthy: false` and the log records `Health check failed: <error>`. An unhealthy provider can starve sentinel run observers while codons using the same key still run because their credential-resolution paths differ. | Check DNS, VPN, and proxy settings. A proxy URL can break sentinel health checks. The internal proxy is disabled by default and enabled with `--proxy`; `--without-proxy` is a parser-real hidden compatibility flag, not a new primary interface.                                                                                                               | [Authentication and models](/operate/authentication-and-models) · [LLM proxy](/reference/llm-proxy)                                                                           |
| `> Stripped ANTHROPIC_API_KEY from process environment (HANKWEAVE_ANTHROPIC_API_KEY=unset)`                                                                                                 | Preflight: environment setup         | The form strips whichever variable is named before provider initialization. When that variable is a proxy variable, the in-process sentinel provider does not inherit that proxy URL; this captured line shows the same mechanism removing `ANTHROPIC_API_KEY`.                                                                                                                                                      | Use the documented `HANKWEAVE_<VAR>=unset` form with the inherited variable's name when it must be removed, then retry the startup check. The captured example removes `ANTHROPIC_API_KEY`; use the same pattern with the proxy variable itself when that proxy is the problem.                                                                                  | [Environment variables](/reference/environment-variables)                                                                                                                     |
| `Budget exceeded for <codonId>: Cost $X exceeded limit $Y`<br />`Duration …s exceeded limit …s`<br />`Output tokens … exceeded limit …`<br />`Context tokens … exceeded limit …`            | Running: budget cap                  | The runtime requests SIGTERM when a cap trips. `onExceeded` determines the result; the default is `"complete"`, even if a required handoff is absent.                                                                                                                                                                                                                                                                | Inspect the artifact before continuing. Use a fail-fast consumer boundary check; choose `"fail"` for a required producer or define an explicit partial-result contract. Calibrate caps from observed work. Editing a saved per-codon cap does not change it on resume.                                                                                           | [Budgets](/concepts/budgets#what-happens-when-a-cap-trips) · [Resume change table](/operate/resume-rollback-and-retry#resume-without-re-running-sealed-work)                  |
| `Module not found` or ENOENT naming a hank-directory script, such as `bun rigs/preflight.ts`                                                                                                | Preflight: rig execution             | Command rigs execute in the execution workspace (`workingDirectory: "project"`), where the hank's own scripts do not exist.                                                                                                                                                                                                                                                                                          | Copy the script into the workspace, then execute the copied path. The `anchor-hank` fixture uses this copy-then-command pattern.                                                                                                                                                                                                                                 | [Rigs](/concepts/rigs)                                                                                                                                                        |
| `Target parent directory does not exist: <path>`                                                                                                                                            | Preparing: rig copy                  | `copy.to` names the full destination, but its parent directory does not exist in the workspace.                                                                                                                                                                                                                                                                                                                      | Put a command such as `"run": "mkdir -p 'pipeline'"` before copying to `pipeline/check.ts`; quote shell path arguments. Validation does not create copy parents. If you changed the rig definition, use a fresh plan rather than expecting resume to adopt it.                                                                                                   | [Copy and command order](/concepts/rigs#what-a-rig-does-before-work-starts) · [Resume change table](/operate/resume-rollback-and-retry#resume-without-re-running-sealed-work) |
| `Query execution failed for codon <id>` with `Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons` | Running: container execution         | The container is running the Claude process as root or with sudo privileges, so the SDK refuses its safety setting and classifies the crash as permanent.                                                                                                                                                                                                                                                            | Run as the image's non-root `node` user (`USER node` in the image or `docker run --user node`). Provide writable output and execution mounts, a read-only source mount, and set `HANKWEAVE_RUNTIME_EXECUTION_BASE_DIR` to the persisted execution mount. Do not weaken the safety check.                                                                         | [Deployment](/operate/deployment)                                                                                                                                             |

Pi gives a busy turn or tool call at least 300 seconds of silence: its busy threshold is `max(shimIdleTimeout, 300s)`. The 120-second fallback above is the idle threshold, not a universal maximum tool duration. An unexplained `SIGKILL` or `SIGTERM` alone does not identify an idle timeout; correlate it with the watchdog, budget, cancellation, and host logs.

```text
[<ts>] [ERROR] Query execution failed for codon summarize-notes
Session ID: N/A
Working directory: /executions/<exec-id>
Error type: Error
Error message: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
Stack trace:
Error: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
    at Rk.getProcessExitError (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:118:11055)
    at ChildProcess.n (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:118:15335)
    at Object.onceWrapper (node:events:634:26)
    at ChildProcess.emit (node:events:531:35)
    at m (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:118:4681)
    at ChildProcess.<anonymous> (file:///usr/local/lib/node_modules/hankweave/node_modules/@anthropic-ai/claude-agent-sdk/sdk.mjs:118:4885)
    at Object.onceWrapper (node:events:634:26)
    at ChildProcess.emit (node:events:531:35)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)
[<ts>] [ERROR] Query error: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons
[<ts>] [ERROR] [CodonRunner] SDK crash classified permanent — routing to exit path for failure policy: Claude Code process exited with code 1. stderr: --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons

exit=1
```

The rig row deserves a closer look, because its fix is a pattern rather than a one-liner. Command rigs execute in the execution workspace, so a script that lives in the hank directory must be copied into the workspace before it can run. The `anchor-hank` fixture's `normalize-aster` rigSetup does exactly this – create a directory, copy the script, then execute the copied path:

```json
      "rigSetup": [
        {
          "type": "command",
          "command": {
            "run": "mkdir -p pipeline"
          }
        },
        {
          "type": "copy",
          "copy": {
            "from": "rigs/preflight.ts",
            "to": "pipeline/preflight.ts"
          }
        },
        {
          "type": "command",
          "command": {
            "run": "bun pipeline/preflight.ts"
          }
        }
      ],
```

For a budget cap, `onExceeded: "complete"` finishes the codon early with a `budget limit reached` info event. `onExceeded: "fail"` fails it without retry and stores a `failureReason` beginning `Budget exceeded: …`. The token fields are `maxOutputTokens` and `maxContextTokens` in `BudgetLimits`.

After a transport or quota recovery and resume, confirm the run actually finished: the journal should show `codon.completed` followed by `RunCompleted`, as in this capture:

```text
{"id": "<id>", "timestamp": "<ts>", "type": "codon.completed", "data": {"codonId": "summarize-notes", "success": true, "cost":"<n>", "duration":"<n>", "exitStatus": {"type": "success"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunCompleted", "runId": "<id>", "transition": {"type": "RunCompleted", "data": {"runId": "<id>"}}, "resultingState": {"currentRunId": null, "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
```

## Separate misleading messages from real failures

Some messages describe a side effect or an earlier cause rather than the failure itself. A budget trip surfaces as a user abort; a telemetry lookup failure prints `ERROR` on a run that succeeded. Match neighboring log lines before choosing a recovery path.

| Symptom                                                                                       | Phase                                 | Cause                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | Fix                                                                                                                                                                                                                                                                                                                  | See                                                                                   |
| --------------------------------------------------------------------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `Claude Code process aborted by user` immediately after a budget trip                         | Running: budget abort                 | Nobody aborted the run. The budget listener sends SIGTERM to the harness, and the SDK surfaces that kill with misleading user-abort wording.                                                                                                                                                                                                                                                                                                                                                      | Correlate the timestamp with `Budget exceeded for <codonId>` and read the codon's stored `failureReason`, whose message is `Budget exceeded: …`. Treat the user-abort line as the harness-side view of the same event.                                                                                               | [Observe and debug](/operate/observe-and-debug) · [Budgets](/concepts/budgets)        |
| `No LLM provider available` from a sentinel report while codons resolve the same provider key | Running: sentinel report              | Sentinel LLM calls are wired only when the registry holds real providers and the sentinel model is a full, registry-known `<provider>/<model>` id. With a partial id such as the captured bare `model: "haiku"`, the call function is a throwing stub and logs four `No LLM provider available` error lines while its codon completes – the first is `[Sentinel:quality-observer] LLM call failed: Error: No LLM provider available` – while a `pi/…` spelling is skipped as not in the registry. | Set a registry-known full sentinel model spelling and a healthy keyed provider. `anthropic/claude-haiku-4-5` and `google/gemini-2.5-flash` fire; `pi/google/gemini-2.5-flash` is skipped with `Model pi/google/gemini-2.5-flash not found in registry`. Keep sentinel key lookup on the `HANKWEAVE_SENTINEL_*` path. | [Authentication and models](/operate/authentication-and-models)                       |
| PostHog `getaddrinfo ENOTFOUND hw-telemetry.southbridge.ai` stack traces in headless stderr   | Completed or running: telemetry flush | This is a telemetry flush failure, not a run failure. Runs complete with exit code `0` around the noise; the defect is unsuppressed `ERROR` output for a transient name-lookup failure.                                                                                                                                                                                                                                                                                                           | Ignore the noise, or opt out with `DO_NOT_TRACK=1` or `HANKWEAVE_TELEMETRY=0`. The precedence is owned by [telemetry](/reference/telemetry).                                                                                                                                                                         | [Telemetry](/reference/telemetry)                                                     |
| `Shutting down server: all codons completed` after a launch that exits `0` without doing work | Completed: silent reuse               | Execution selection matched the data signature to an existing `RunCompleted` execution and silently resumed it without naming the execution. The second launch logs `Shutting down server: all codons completed`, then `Shutdown: all codons completed (exit code: 0)`, and its event journal gains no new codon events (`delta 0`) – nothing re-ran.                                                                                                                                             | Relaunch with `--start-new` (`-n`) to force a fresh execution. Check for a new execution directory and codon activity before trusting exit code `0`.                                                                                                                                                                 | [Runbook](/operate/runbook)                                                           |
| `0 system prompts` in a `GOOD TO RUN!` summary despite configured system instructions         | Before-run: validation summary        | The label counts codon `appendSystemPromptFile` files, not inline append text or the global system prompt.                                                                                                                                                                                                                                                                                                                                                                                        | Check `globalSystemPromptFile`/`globalSystemPromptText`, `appendSystemPromptText`, and the separate global-setting indicator. Do not add duplicate instructions merely to increase this count.                                                                                                                       | [Prompt wiring and counts](/author/prompts#wire-prompt-files-and-validate-the-counts) |

> **Pitfall:** Exit code `0` does not prove that your hank ran. A silent resume of a completed execution can log “all codons completed” and do nothing; check for `Created new execution directory` or codon activity before trusting the green exit.

After the `--start-new` relaunch, look for the captured new-directory line and the fresh startup block – their presence is what distinguishes a real fresh run from a silent reuse:

```text
Created new execution directory: ~/.hankweave-executions/<exec-id>
New execution: <exec-id>
  Source → data
  Exec   → ~/.hankweave-executions/<exec-id>
  SDKs   → Claude node_modules ✓

╭──────────────────────────────────────────────────────────────────────────────╮
│  Minimal single provider v1.0.0                                              │
  …
```

## Escalate symptoms this table does not cover

If the message is too vague to diagnose, we'll gather the evidence in this order:

1. Inspect `state.json`.
2. Read the exact `claudeLogPath` stored there.
3. Read the journal.
4. Diff the checkpoint – a run-boundary marker.

For failure classes, `failureReason`, and exit codes, use [errors and exit codes](/reference/errors-and-exit-codes). For retry, resume, rollback, and replay, use [resume, rollback, and retry](/operate/resume-rollback-and-retry). Use [observe and debug](/operate/observe-and-debug) for the artifact sequence and [help](/help) for the great-bug-report recipe.

If an authentication or model error survives editing `hank.json`, check whether it comes from the wizard, a template, or the pinned health-check model. The separate model-selection paths are explained in [authentication and models](/operate/authentication-and-models).