Errors and exit codes

When something goes wrong in a Hankweave run, several different mechanisms decide what happens next: a classifier that reads error text and decides whether a failure is worth retrying, a failureReason object that records the verdict, severity levels on error events, and finally the process exit code your shell or CI system sees. This page is the lookup reference for all of them. Each section defines one surface, states its exact values, and notes the version in which the behavior appeared, so you can go from an observed failure to the right table without reading the whole page.

Error classifier table#

A codon is one agent task in the run sequence. When a codon fails and there is error text to inspect, the classifier evaluates that text against an ordered list of patterns with first-match-wins semantics: if the text matches two rows, classification stops at the first ordered match. matchOrder is that evaluation order, running from 1 through 20; it is not alphabetical order, and the table below is sorted by it.

Evaluation order: explicit-rate-limit, billing-quota, bedrock-sts-assume-role, bedrock-access-denied, bedrock-invalid-credentials, bedrock-signature, bedrock-expired-token, bedrock-sso-session, bedrock-no-credentials, bedrock-pi-not-configured, bedrock-on-demand-throughput, bedrock-model-not-found, timeout, auth, invalid-request, bare-429, transport, server-error, fallback-no-session, fallback-default.

The table exposes each classification ID, its triggering pattern, the result type, the retriable boolean, and the evaluation order. A retriable verdict may also carry retryAfterMs; that provider-supplied hint belongs to the failure details described in the next section, not to this table.

Bedrock permanent failures (since 0.9.0)#

The Bedrock permanent-failure family occupies matchOrder 3–12, so these patterns are checked before the generic timeout, auth, and transport rows. The table below retains declaration order and pairs each AWS error name or pattern with the remedy for that specific failure, since the fix differs depending on which credential, permission, or model-availability problem Bedrock reported.

What a failureReason records#

The classifier's verdict is stored in a failureReason object: the category and retry result for a failed codon, with optional human-readable detail and provider retry timing. The 20 classifier IDs above are labels for matching branches; they are not additional failureReason.type enum members. The schema below is the authoritative shape.

const failureReasonSchema = z.object({
  type: z.enum(["timeout", "rate-limit", "api-error", "sentinel-load-failure", "unknown"]),
  retriable: z.boolean(),
  message: z.string().optional(),
  sentinelRefs: z.array(z.string()).optional(), // Which sentinels failed (for sentinel-load-failure)
  /**
   * How long the provider asked us to wait before retrying, in milliseconds,
   * when its error carried an explicit hint (a Retry-After header value, a
   * "retry after 30s" phrasing, etc). Parsed by classifyApiErrorText and
   * consumed by computeRetryDelayMs, which prefers it over computed backoff —
   * the provider knows its own limit window better than we can guess.
   */
  retryAfterMs: z.number().optional(),
});

Two enum values need qualification. sentinel-load-failure is constructed directly for a sentinel failure when failCodonIfNotLoaded=true; it is not returned by classifyApiErrorText and is not retriable. unknown is retained in the schema (the event-shape definition), but no classifyApiErrorText return site emits it. The generated classifier taxonomy therefore supersedes older five-value failureReason lists elsewhere in the documentation.

Retriable versus permanent failures#

The retriable flag is the decision that matters operationally, because it determines whether the runtime attempts the codon again.

Retriable: API-level failures such as connection resets, timeouts, 429 rate limits, and 5xx or overloaded responses.

Permanent: Authentication, invalid requests, billing, credit or quota exhaustion, and usage-limit caps. This doctrine changed in 0.8.0 and applies uniformly to the harnesses.

Unrecognized error text is retriable by default. The asymmetry is deliberate: one incorrect retriable verdict costs one bounded attempt, while one incorrect permanent verdict can end the whole run. When sessionEstablished=false, an unmatched local setup failure is permanent; unless supplied otherwise, sessionEstablished defaults to true.

retryAfterMs is provider-supplied timing parsed from headers, worded forms, or RFC 9110 delay-seconds. The runtime prefers it over computed exponential backoff because the provider knows its limit window better than the runtime can infer it. It is present only on retriable verdicts, so there is no retry to schedule for a permanent verdict.

failedDuring execution phases#

Beyond why a codon failed, the state record says how far it got. A failed codon records the execution phase in failedDuring, and the supported phase values are preparing, starting, initializing, running, and completing-sentinels.

  /**
   * Which state we were in when failure occurred.
   * Helps understand how far we got.
   *
   * Used by: Error analysis, retry strategies
   * Example: "preparing" means rig setup failed
   * Note: Can include "completing-sentinels" if checkpoint creation fails during that codon
   */
  failedDuring: "preparing" | "starting" | "initializing" | "running" | "completing-sentinels";

Error severity levels#

An error event may carry one of four severity levels, ranging from a log-only warning to a server shutdown. The field is optional; when omitted, the event carries no severity.

Harness self-test failure classes#

Not every failure passes through the codon retry loop. A harness (the model-session runner) self-test is pre-run validation for each model, and these failures gate startup, so they never reach the classifier at all. An in-band check is reported inside the harness's own result; agent.found is the result field used by the binary-missing classification. The table maps each category to the guidance shown to the user.

Self-test category classification has been available since 0.7.0. --validate and preflight run self-tests before the run; the guidance routes the user to the relevant fix instead of defaulting every failure to an API-key problem.

Exit code contract#

The process exit code is what CI and shell scripts observe. It describes the run or the CLI path as a whole, and it is separate from a codon's internal exitStatus. A client request is a graceful-shutdown reason; server.force_shutdown is the force-shutdown command.

An explicit exitCode override bypasses this computed contract. Without an override, the code is computed from the shutdown reason. Bare invocation, --version, --help, --init, --validate, and --cleanup exit before (or without) starting the server; only the Run path computes its exit code through computeExitCode in shutdown(). The next table covers those CLI paths individually.

A remote hank is a workflow definition fetched from a GitHub, GitLab, or Bitbucket URL over HTTPS or SSH and cached locally. The CLI exits 1 when that fetch fails.

Shutdown watchdog and force-stop#

Shutdown itself has three possible paths, and they differ in what happens to checkpoints, telemetry, and the final exit code.

The observed double-SIGINT capture ends with exit=0: graceful SIGINT shutdown completed before the second signal's forceShutdown escalation ran. Treat that as a race limit of this capture, not as proof that forceShutdown completed or as an unconditional second-signal exit of 1.

For an interrupted shutdown, inspect the checkpoint commit's first line: it uses the exit:<codonId> [run:…] <codonName> prefix. The checkpoint-created state/event field maps exit to checkpointType: "skipped", because only rig-setup, completed, and error receive their same-named types. The Git commit prefix, the checkpointType field, and the process exit code are three distinct signals; do not use one as a substitute for another.

Since 0.7.3, the shutdown watchdog arms on every shutdown() call that exits the process, including second-signal escalation to forceShutdown. The watchdog backstop is SHUTDOWN_WATCHDOG_MS: 30000; the cleanup delay after a normal exit is 100 ms.

outputFiles can run beforeCopy commands before copying. A failure in either the before-copy operation or the copy operation fails the run.

onFailure: "abort" selects abort behavior. In --headless mode, since 0.7.3, a retriable failure fails fast with exit code 1 instead of waiting for an interactive client to initiate a retry.

Budget metering is usage-time, not a prepaid guarantee. Each assistant message is priced and added to the running total; when BudgetTracker.addCost reaches maxDollars, it emits exceeded and the codon runner requests SIGTERM, so an already-billable message may push usage to or past the limit. The interrupt and onExceeded policy are separate: onExceeded: "complete" is the default and finalizes the interrupted codon as completed with partial output (run exit 0); "fail" finalizes it as failed (exit 1) and synthesizes failureReason {type: "unknown", retriable: false, message: "Budget exceeded: …"}. A codon.completed event may also carry budgetExceeded: {currency, limit, used}. Budget exhaustion is not a loop.iteration.completed.terminationReason; that field allows only iteration_limit, context_exceeded, sentinel_skip, or failure. Budget exhaustion is never classified via classifyApiErrorText.

Codon-level exit information#

The process-level contract above is not the only place exit information appears. A codon's completion event and its state record expose different fields, and the shapes differ between them.

When a failed codon has no result and no crash text, the runtime synthesizes a failure reason. It is retriable when sessionEstablished !== false; when that setting is unknown, it defaults to true. Synthesis returns undefined for force-stop and context-exceeded. context-exceeded means that the model's context window was exceeded.

sentinel-load-failure is constructed directly for a sentinel failure when failCodonIfNotLoaded=true, with retriable: false and sentinelRefs. failCodonIfNotLoaded is a codon-level setting that defaults to false. The failed codon's state record carries the recorded per-codon process exit code as a plain number, such as 1, -1, or 130; it is not the run-exit override. Only the optional exitCode parameter of shutdown()/computeExitCode overrides the computed exit contract.