Errors and exit codes

When something goes wrong in a Hankweave run, several different mechanisms decide what happens next: a classifier that reads error text and decides whether a failure is worth retrying, a failureReason object that records the verdict, severity levels on error events, and finally the process exit code your shell or CI system sees. This page is the lookup reference for all of them. Each section defines one surface, states its exact values, and notes the version in which the behavior appeared, so you can go from an observed failure to the right table without reading the whole page.

Error classifier table#

A codon is one agent task in the run sequence. When a codon fails and there is error text to inspect, the classifier evaluates that text against an ordered list of patterns with first-match-wins semantics: if the text matches two rows, classification stops at the first ordered match. matchOrder is that evaluation order, running from 1 through 20; it is not alphabetical order, and the table below is sorted by it.

Evaluation order: explicit-rate-limit, billing-quota, bedrock-sts-assume-role, bedrock-access-denied, bedrock-invalid-credentials, bedrock-signature, bedrock-expired-token, bedrock-sso-session, bedrock-no-credentials, bedrock-pi-not-configured, bedrock-on-demand-throughput, bedrock-model-not-found, timeout, auth, invalid-request, bare-429, transport, server-error, fallback-no-session, fallback-default.

The table exposes each classification ID, its triggering pattern, the result type, the retriable boolean, and the evaluation order. A retriable verdict may also carry retryAfterMs; that provider-supplied hint belongs to the failure details described in the next section, not to this table.

Bedrock permanent failures#

The Bedrock permanent-failure family occupies matchOrder 3–12, so these patterns are checked before the generic timeout, auth, and transport rows. The table below retains declaration order and pairs each AWS error name or pattern with the remedy for that specific failure, since the fix differs depending on which credential, permission, or model-availability problem Bedrock reported. [since 0.9.0] The Bedrock permanent-failure family was added in 0.9.0.

What a failureReason records#

The classifier's verdict is stored in a failureReason object: the category and retry result for a failed codon, with optional human-readable detail and provider retry timing. The 20 classifier IDs above are labels for matching branches; they are not additional failureReason.type enum members. The schema below is the authoritative shape.

const failureReasonSchema = z.object({
  type: z.enum(["timeout", "rate-limit", "api-error", "sentinel-load-failure", "unknown"]),
  retriable: z.boolean(),
  message: z.string().optional(),
  sentinelRefs: z.array(z.string()).optional(), // Which sentinels failed (for sentinel-load-failure)
  /**
   * How long the provider asked us to wait before retrying, in milliseconds,
   * when its error carried an explicit hint (a Retry-After header value, a
   * "retry after 30s" phrasing, etc). Parsed by classifyApiErrorText and
   * consumed by computeRetryDelayMs, which prefers it over computed backoff —
   * the provider knows its own limit window better than we can guess.
   */
  retryAfterMs: z.number().optional(),
});

Two enum values need qualification. sentinel-load-failure is constructed directly for a sentinel failure when failCodonIfNotLoaded=true; it is not returned by classifyApiErrorText and is not retriable. unknown is retained in the schema (the event-shape definition), but no classifyApiErrorText return site emits it. The generated classifier taxonomy therefore supersedes older five-value failureReason lists elsewhere in the documentation.

Retriable versus permanent failures#

The retriable flag is the decision that matters operationally, because it determines whether the runtime attempts the codon again.

Retriable: API-level failures such as connection resets, timeouts, 429 rate limits, and 5xx or overloaded responses.

Permanent: Authentication, invalid requests, billing, credit or quota exhaustion, and usage-limit caps. [changed 0.8.0] The retriable-versus-permanent doctrine was overhauled in 0.8.0 and applies uniformly to the harnesses.

Unrecognized error text is retriable by default. The asymmetry is deliberate: one incorrect retriable verdict costs one bounded attempt, while one incorrect permanent verdict can end the whole run. When sessionEstablished=false, an unmatched local setup failure is permanent; unless supplied otherwise, sessionEstablished defaults to true.

retryAfterMs is provider-supplied timing parsed from headers, worded forms, or RFC 9110 delay-seconds. The runtime prefers it over computed exponential backoff because the provider knows its limit window better than the runtime can infer it. It is present only on retriable verdicts, so there is no retry to schedule for a permanent verdict.

failedDuring execution phases#

Beyond why a codon failed, the state record says how far it got. A failed codon records the execution phase in failedDuring, and the supported phase values are preparing, starting, initializing, running, and completing-sentinels.

  /**
   * Which state we were in when failure occurred.
   * Helps understand how far we got.
   *
   * Used by: Error analysis, retry strategies
   * Example: "preparing" means rig setup failed
   * Note: Can include "completing-sentinels" if checkpoint creation fails during that codon
   */
  failedDuring: "preparing" | "starting" | "initializing" | "running" | "completing-sentinels";

The comment inside that declaration claims completing-sentinels can record a checkpoint-creation failure. It does not match the code: when checkpoint creation fails, the codon's failedDuring comes from a codon record read before the move to completing-sentinels, so it holds the earlier status, normally running. failedDuring: "completing-sentinels" is written only when a failure reads the live status during the sentinel drain, which happens on a force stop or a crash recovery of a run left in that phase.

Error severity levels#

An error event may carry one of four severity levels, ranging from a log-only warning to a server shutdown. The field is optional; when omitted, the event carries no severity.

Harness self-test failure classes#

Not every failure passes through the codon retry loop. A harness (the model-session runner) self-test is pre-run validation for each model, and these failures gate startup, so they never reach the classifier at all. An in-band check is reported inside the harness's own result; agent.found is the result field used by the binary-missing classification. The table maps each category to the guidance shown to the user.

One row in that table is effectively unreachable: the auth class requires a failed check named api_key, and no shipped harness emits a check with that name (the Claude Agent SDK and Pi self-tests name their key check authentication). A missing key is therefore classed check, the auth guidance line is not added, and No authentication found (set ANTHROPIC_API_KEY) is the failed authentication check's own message. That check is a credential-presence test, not a billable provider health check.

[since 0.7.0] Self-test category classification was added in 0.7.0. --validate and preflight run the same per-model self-tests before the run; the guidance routes the user to the relevant fix instead of defaulting every failure to an API-key problem.

Exit code contract#

The process exit code is what CI and shell scripts observe. It describes the run or the CLI path as a whole, and it is separate from a codon's internal exitStatus.

The exit code follows the shutdown reason. It is 0 only for all codons completed when the run did not fail, and for the exact graceful reasons SIGINT, SIGTERM, and client request. Every other reason exits 1. A server.shutdown command without data.reason uses client request (exit 0), while one that carries its own data.reason uses that string and exits 1. A single q or Ctrl+C in the built-in TUI shuts down with reason user request, which also exits 1.

An explicit exitCode override bypasses this computed contract. Without an override, the code is computed from the shutdown reason. Bare invocation, --version, --help, --init, --validate, and --cleanup exit before (or without) starting the server; only the Run path computes its exit code through computeExitCode in shutdown(). The next table covers those CLI paths individually.

A remote hank is a workflow definition fetched from a GitHub, GitLab, or Bitbucket URL over HTTPS or SSH and cached locally. The CLI exits 1 when that fetch fails.

Exit code 0 is not a full CI verdict on its own. It reports the runtime's own outcome; it does not prove the produced artifacts are semantically correct. A CI gate that cares about the work should check the artifacts explicitly rather than treat exit 0 as success.

Shutdown watchdog and force-stop#

Shutdown itself has three possible paths, and they differ in what happens to checkpoints, telemetry, and the final exit code.

The watchdog bounds each shutdown window: it force-exits with the same code computeExitCode would produce if a graceful shutdown does not complete within 30 seconds, and the runtime re-arms it after the synchronous trace upload, so a whole shutdown can exceed 30 seconds. A run that never begins shutting down, such as a codon running past its time cap, has no watchdog; the backstop does not stop a stuck run.

The double-SIGINT capture ends with exit=1. Its server log shows the force path completing: Shutting down server: SIGINT, Killing current codon runner for shutdown, Force shutting down server: second SIGINT, Force shutdown complete. The second signal reached forceShutdown before the graceful shutdown finished.

Force shutdown aborts the running harness session rather than killing a process with SIGKILL: for the Claude Agent SDK it is an abort that ends the agent subprocess with SIGTERM, and for the embedded Pi SDK it is an in-process session abort. It skips checkpoints, telemetry, and state transitions, and exits 1.

An exit: checkpoint commit is written only by a graceful shutdown(), when a codon is still in progress and the reason is not all codons completed: a first SIGINT or SIGTERM, a client server.shutdown, or a codon-failure shutdown. forceShutdown, a second signal, a crash, or a kill writes none. Inspect the commit's first line for the exit:<codonId> [run:…] <codonName> prefix. The checkpoint-created state/event field maps exit to checkpointType: "skipped", because only rig-setup, completed, and error receive their same-named types. The Git commit prefix, the checkpointType field, and the process exit code are three distinct signals; do not use one as a substitute for another. Even on the graceful path, a failed exit checkpoint is only logged (Exit checkpoint skipped: …).

[since 0.7.3] The shutdown watchdog arms on every shutdown() call that exits the process, including second-signal escalation to forceShutdown. The watchdog backstop is SHUTDOWN_WATCHDOG_MS: 30000; the cleanup delay after a normal exit is 100 ms.

outputFiles can run beforeCopy commands before copying. A failure in either the before-copy operation or the copy operation fails the run.

onFailure: "abort" selects abort behavior. [since 0.7.3] In --headless mode, a retriable failure under onFailure: "abort" fails fast with exit code 1 instead of waiting for an interactive client to initiate a retry.

Budget metering is usage-time, not a prepaid guarantee. Each assistant message is priced and added to the running total; when BudgetTracker.addCost reaches maxDollars, it emits exceeded and the codon runner requests the codon's interruption with SIGTERM, so an already-billable message may push usage to or past the limit. The interrupt and the onExceeded policy are separate, and both policies interrupt the codon: onExceeded: "complete" is the default and finalizes the interrupted codon as completed (run exit 0), while "fail" finalizes it as failed (exit 1) and synthesizes failureReason {type: "unknown", retriable: false, message: "Budget exceeded: …"}. A codon.completed event may carry budgetExceeded: {currency, limit, used} whenever a cap tripped, on a completed or failed codon alike. Budget exhaustion has no loop.iteration.completed.terminationReason value: that field is optional, the runtime sets it only on a loop's final iteration and only to context_exceeded or iteration_limit, and sentinel_skip and failure are schema-allowed but never emitted. Budget exhaustion is never classified via classifyApiErrorText.

Codon-level exit information#

The process-level contract above is not the only place exit information appears. A codon's completion event and its state record expose different fields, and the shapes differ between them.

When a failed codon has no result and no crash text, the runtime synthesizes a failure reason. It is retriable when sessionEstablished !== false; when that setting is unknown, it defaults to true. Synthesis returns undefined for force-stop and context-exceeded. context-exceeded means that the model's context window was exceeded.

sentinel-load-failure is constructed directly for a sentinel config-loading failure when failCodonIfNotLoaded=true, with retriable: false and sentinelRefs. The flag covers only config-loading failures: a missing, unparseable or schema-invalid config file or ref, or an exception out of the sentinel manager (fatal then only when every loaded config sets the flag). A sentinel skipped because its model is not in the registry, its provider is unconfigured or unhealthy, or it hit a per-sentinel instantiation error is only logged, and the codon runs without it even with the flag set.

The codon state record's exitCode is a plain number: 0 or 1 from the harness, or -1 written by the runtime for a force stop, a run crash, or a failure before any runner ran (such as a sentinel load failure or a dispatch error). Nothing writes 130, and -1 does not mean a signal kill; after a graceful Ctrl+C the interrupted codon stays running with a null exit code. It is not the run-exit override: only the optional exitCode parameter of shutdown()/computeExitCode overrides the computed exit contract.