# Errors and exit codes When something goes wrong in a Hankweave run, several different mechanisms decide what happens next: a classifier that reads error text and decides whether a failure is worth retrying, a `failureReason` object that records the verdict, severity levels on error events, and finally the process exit code your shell or CI system sees. This page is the lookup reference for all of them. Each section defines one surface, states its exact values, and notes the version in which the behavior appeared, so you can go from an observed failure to the right table without reading the whole page. ## Error classifier table A codon is one agent task in the run sequence. When a codon fails and there is error text to inspect, the classifier evaluates that text against an ordered list of patterns with first-match-wins semantics: if the text matches two rows, classification stops at the first ordered match. `matchOrder` is that evaluation order, running from 1 through 20; it is not alphabetical order, and the table below is sorted by it. **Evaluation order:** `explicit-rate-limit`, `billing-quota`, `bedrock-sts-assume-role`, `bedrock-access-denied`, `bedrock-invalid-credentials`, `bedrock-signature`, `bedrock-expired-token`, `bedrock-sso-session`, `bedrock-no-credentials`, `bedrock-pi-not-configured`, `bedrock-on-demand-throughput`, `bedrock-model-not-found`, `timeout`, `auth`, `invalid-request`, `bare-429`, `transport`, `server-error`, `fallback-no-session`, `fallback-default`. The table exposes each classification ID, its triggering pattern, the result type, the retriable boolean, and the evaluation order. A retriable verdict may also carry `retryAfterMs`; that provider-supplied hint belongs to the failure details described in the next section, not to this table. | id | pattern | type | retriable | matchOrder | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | --------- | ---------: | | `explicit-rate-limit` | `rate limit` \| `rate_limit` \| `rate-limit` (substring includes, lowercased) | `rate-limit` | `true` | 1 | | `billing-quota` | `credit` \| `billing` \| `insufficient` \| `quota` \| `available balance` \| `out of budget` \| `/usage\s?limit/` | `api-error` | `false` | 2 | | `bedrock-sts-assume-role` | `sts:assumerole` AND (`accessdenied` OR `not authorized`) | `api-error` | `false` | 3 | | `bedrock-access-denied` | `accessdeniedexception` | `api-error` | `false` | 4 | | `bedrock-invalid-credentials` | `unrecognizedclientexception` OR `security token included in the request is invalid` | `api-error` | `false` | 5 | | `bedrock-signature` | `invalidsignatureexception` OR `signaturedoesnotmatch` OR `signature we calculated does not match` | `api-error` | `false` | 6 | | `bedrock-expired-token` | `expiredtokenexception` | `api-error` | `false` | 7 | | `bedrock-sso-session` | `sso session` AND (`not found` OR `invalid` OR `expired`) | `api-error` | `false` | 8 | | `bedrock-no-credentials` | `could not load credentials` OR `unable to locate credentials` | `api-error` | `false` | 9 | | `bedrock-pi-not-configured` | `provider is not configured: amazon-bedrock` | `api-error` | `false` | 10 | | `bedrock-on-demand-throughput` | `/on-demand throughput isn.t supported/` | `api-error` | `false` | 11 | | `bedrock-model-not-found` | `resourcenotfoundexception` OR `the provided model identifier is invalid` | `api-error` | `false` | 12 | | `timeout` | `timeout` OR `timed out` | `timeout` | `true` | 13 | | `auth` | `\b401\b` OR `\b403\b` OR `authentication` OR `unauthorized` OR `forbidden` OR `api key` OR `oauth` | `api-error` | `false` | 14 | | `invalid-request` | `invalid_request` OR `invalid request` OR `validationexception` OR `validation error:` OR `prompt is too long` OR `\b400\b` | `api-error` | `false` | 15 | | `bare-429` | `\b429\b` | `rate-limit` | `true` | 16 | | `transport` | `socket` OR `econnreset` OR `econnrefused` OR `econnaborted` OR `epipe` OR `etimedout` OR `enotfound` OR `eai_again` OR `unable to connect` OR `fetch failed` OR `connection reset` OR `connection closed` OR `connection error` OR (`stream` AND `terminated`) OR `network` | `api-error` | `true` | 17 | | `server-error` | `\b500\b` OR `\b502\b` OR `\b503\b` OR `\b504\b` OR `\b529\b` OR `overloaded` OR `internal server error` OR `bad gateway` OR `service unavailable` | `api-error` | `true` | 18 | | `fallback-no-session` | `*` (no prior branch matched) AND `opts.sessionEstablished === false` | `api-error` | `false` | 19 | | `fallback-default` | `*` (no prior branch matched) | `api-error` | `true` | 20 | ### Bedrock permanent failures (since 0.9.0) The Bedrock permanent-failure family occupies `matchOrder` 3–12, so these patterns are checked before the generic timeout, auth, and transport rows. The table below retains declaration order and pairs each AWS error name or pattern with the remedy for that specific failure, since the fix differs depending on which credential, permission, or model-availability problem Bedrock reported. | matchOrder | AWS error name / pattern | classification ID | retriable | remedy | | ---------: | ------------------------------------------------------ | ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 3 | `sts:assumerole` access denied / not authorized | `bedrock-sts-assume-role` | false | The configured role can't be assumed. Check the profile's role\_arn, the role's trust policy (does it trust your source identity?), and any external\_id/MFA requirement. | | 4 | `AccessDeniedException` | `bedrock-access-denied` | false | AWS credentials can't invoke this model in this region. Check: (a) IAM policy / SCP restrictions – cross-region 'us.' inference profiles need invoke permission on the profile AND its underlying foundation models (simplest: allow bedrock:InvokeModel\* on Resource '\*' for this principal); (b) first-time Anthropic use requires the use-case form – open the model once in the Bedrock console playground; (c) Marketplace-served models need one first invoke by a user with AWS Marketplace permissions. | | 5 | `UnrecognizedClientException` / invalid security token | `bedrock-invalid-credentials` | false | AWS credentials are invalid. Check AWS\_BEARER\_TOKEN\_BEDROCK / AWS\_ACCESS\_KEY\_ID + AWS\_SECRET\_ACCESS\_KEY. Bedrock short-term API keys expire – generate a long-term key (AWS Console → Bedrock → API keys). | | 6 | `InvalidSignatureException` / signature mismatch | `bedrock-signature` | false | Request signing failed – AWS\_SECRET\_ACCESS\_KEY doesn't match AWS\_ACCESS\_KEY\_ID (typo, stale copy, or wrong account), or the system clock is badly skewed. Re-copy the secret for this access key or generate a fresh key pair. | | 7 | `ExpiredTokenException` | `bedrock-expired-token` | false | Re-run `aws sso login --profile ` or refresh the temporary credentials. | | 8 | invalid or expired SSO session | `bedrock-sso-session` | false | Re-run `aws sso login --profile ` to refresh the Identity Center token cache. | | 9 | credentials could not be loaded or located | `bedrock-no-credentials` | false | No AWS credentials found. Quickest: set AWS\_BEARER\_TOKEN\_BEDROCK (AWS Console → Bedrock → API keys → long-term key). Enterprise: set AWS\_PROFILE after `aws sso login`, or AWS\_ACCESS\_KEY\_ID + AWS\_SECRET\_ACCESS\_KEY. Also set AWS\_REGION. | | 10 | provider is not configured: `amazon-bedrock` | `bedrock-pi-not-configured` | false | No AWS credentials visible to the embedded pi runtime. Set AWS\_BEARER\_TOKEN\_BEDROCK (AWS Console → Bedrock → API keys → long-term key), AWS\_PROFILE after `aws sso login`, or AWS\_ACCESS\_KEY\_ID + AWS\_SECRET\_ACCESS\_KEY. Also set AWS\_REGION. An on-disk default profile or instance role alone is not detected on this route. | | 11 | on-demand throughput is unsupported | `bedrock-on-demand-throughput` | false | This model needs an inference profile on Bedrock. Prefix the model id with your region group's cross-region profile prefix – 'us.', 'eu.', 'jp.', 'au.', or 'us-gov.' – or use a 'global.' profile where available. | | 12 | invalid model identifier / resource not found | `bedrock-model-not-found` | false | Model not found in this region. Set AWS\_REGION to a region that serves it, or use the 'global.' inference profile id. | > **DeepDive:** The order has deliberate traps: rate-limit words beat billing, billing beats bare-429, Bedrock patterns beat timeout, timeout beats status codes, and auth beats invalid-request. A rate-limit error mentioning quota is transient; a permanent insufficient-quota 429 is billing; an AWS signature dump containing a timeout header remains a Bedrock credential failure. ## What a failureReason records The classifier's verdict is stored in a `failureReason` object: the category and retry result for a failed codon, with optional human-readable detail and provider retry timing. The 20 classifier IDs above are labels for matching branches; they are not additional `failureReason.type` enum members. The schema below is the authoritative shape. ```ts const failureReasonSchema = z.object({ type: z.enum(["timeout", "rate-limit", "api-error", "sentinel-load-failure", "unknown"]), retriable: z.boolean(), message: z.string().optional(), sentinelRefs: z.array(z.string()).optional(), // Which sentinels failed (for sentinel-load-failure) /** * How long the provider asked us to wait before retrying, in milliseconds, * when its error carried an explicit hint (a Retry-After header value, a * "retry after 30s" phrasing, etc). Parsed by classifyApiErrorText and * consumed by computeRetryDelayMs, which prefers it over computed backoff — * the provider knows its own limit window better than we can guess. */ retryAfterMs: z.number().optional(), }); ``` | Field | Shape | Meaning | | -------------- | ------------------------------------------------------------------------ | ----------------------------------------------- | | `type` | `timeout \| rate-limit \| api-error \| sentinel-load-failure \| unknown` | Failure category. | | `retriable` | boolean | Whether the failure may be retried. | | `message` | optional string | Human-readable detail when supplied. | | `sentinelRefs` | optional `string[]` | References for `sentinel-load-failure`. | | `retryAfterMs` | optional number | Provider-supplied retry timing in milliseconds. | Two enum values need qualification. `sentinel-load-failure` is constructed directly for a sentinel failure when `failCodonIfNotLoaded=true`; it is not returned by `classifyApiErrorText` and is not retriable. `unknown` is retained in the schema (the event-shape definition), but no `classifyApiErrorText` return site emits it. The generated classifier taxonomy therefore supersedes older five-value `failureReason` lists elsewhere in the documentation. ## Retriable versus permanent failures The `retriable` flag is the decision that matters operationally, because it determines whether the runtime attempts the codon again. **Retriable:** API-level failures such as connection resets, timeouts, 429 rate limits, and 5xx or overloaded responses. **Permanent:** Authentication, invalid requests, billing, credit or quota exhaustion, and usage-limit caps. This doctrine changed in 0.8.0 and applies uniformly to the harnesses. Unrecognized error text is retriable by default. The asymmetry is deliberate: one incorrect retriable verdict costs one bounded attempt, while one incorrect permanent verdict can end the whole run. When `sessionEstablished=false`, an unmatched local setup failure is permanent; unless supplied otherwise, `sessionEstablished` defaults to true. `retryAfterMs` is provider-supplied timing parsed from headers, worded forms, or RFC 9110 delay-seconds. The runtime prefers it over computed exponential backoff because the provider knows its limit window better than the runtime can infer it. It is present only on retriable verdicts, so there is no retry to schedule for a permanent verdict. ## failedDuring execution phases Beyond why a codon failed, the state record says how far it got. A failed codon records the execution phase in `failedDuring`, and the supported phase values are `preparing`, `starting`, `initializing`, `running`, and `completing-sentinels`. ```ts /** * Which state we were in when failure occurred. * Helps understand how far we got. * * Used by: Error analysis, retry strategies * Example: "preparing" means rig setup failed * Note: Can include "completing-sentinels" if checkpoint creation fails during that codon */ failedDuring: "preparing" | "starting" | "initializing" | "running" | "completing-sentinels"; ``` ## Error severity levels An error event may carry one of four severity levels, ranging from a log-only warning to a server shutdown. The field is optional; when omitted, the event carries no severity. | Level | Effect | | ----------- | -------------------------------------------------- | | `fatal` | Shuts down the server. | | `codon` | Fails the current codon while the server survives. | | `operation` | Records one failed operation. | | `warning` | Is logged only. | ## Harness self-test failure classes Not every failure passes through the codon retry loop. A harness (the model-session runner) self-test is pre-run validation for each model, and these failures gate startup, so they never reach the classifier at all. An in-band check is reported inside the harness's own result; `agent.found` is the result field used by the binary-missing classification. The table maps each category to the guidance shown to the user. | Category | Classification | Guidance | | ---------------- | ---------------------------------------------- | ---------------------------------------------------------------------- | | `launch` | The harness crashed before producing a result. | “The harness failed to start. See the captured output above.” | | `auth` | An in-band failed check is named `api_key`. | “Ensure the required API keys / credentials are configured correctly.” | | `binary-missing` | An in-band result has no `agent.found`. | “The agent for one or more models could not be located…” | | `check` | An in-band check failed for another reason. | Per-model messages only. | | `unknown` | The failure could not be classified. | Per-model messages only. | Self-test category classification has been available since 0.7.0. `--validate` and preflight run self-tests before the run; the guidance routes the user to the relevant fix instead of defaulting every failure to an API-key problem. ## Exit code contract The process exit code is what CI and shell scripts observe. It describes the run or the CLI path as a whole, and it is separate from a codon's internal `exitStatus`. A client request is a graceful-shutdown reason; `server.force_shutdown` is the force-shutdown command. | Exit code | Conditions | | --------: | --------------------------------------------------------------------------------------------------- | | `0` | All codons completed, or shutdown was graceful because of `SIGINT`, `SIGTERM`, or a client request. | | `1` | A codon failed, the process crashed, or shutdown had an unexpected reason. | An explicit `exitCode` override bypasses this computed contract. Without an override, the code is computed from the shutdown reason. Bare invocation, `--version`, `--help`, `--init`, `--validate`, and `--cleanup` exit before (or without) starting the server; only the Run path computes its exit code through `computeExitCode` in `shutdown()`. The next table covers those CLI paths individually. | CLI path | Exit code | Condition | | --------------------------------- | --------: | ------------------------------------------------------------- | | Parse | `1` | Argument parsing fails. | | Wizard | `0` | Bare invocation completes. | | `--version` / `--help` | `0` | The early-exit request completes. | | `--init` | `0` / `1` | Success / failure. | | `--validate` | `0` / `1` | Success / failure. | | `--cleanup` | `0` / `1` | Success / failure. | | Attach | `1` | The lock-file read fails. | | Server startup or execution setup | `1` | Startup or setup fails. | | Remote hank fetch | `1` | Fetch fails. | | Replay | `1` | Replay is mutually exclusive with execution, or replay fails. | | Run | `0` / `1` | Completion / codon failure, crash, or unexpected shutdown. | | Unhandled `main()` throw | `1` | The CLI's main function throws without handling the failure. | A remote hank is a workflow definition fetched from a GitHub, GitLab, or Bitbucket URL over HTTPS or SSH and cached locally. The CLI exits 1 when that fetch fails. ### Shutdown watchdog and force-stop Shutdown itself has three possible paths, and they differ in what happens to checkpoints, telemetry, and the final exit code. | Mechanism | Exit behavior | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Graceful shutdown | Uses the `computeExitCode` result for the shutdown reason. | | Shutdown watchdog | After 30 seconds, force-exits with the same code that `computeExitCode` would produce if graceful shutdown does not complete. This prevents a wedged process from hanging CI. | | Force shutdown | The `forceShutdown` path schedules exit code `1` and skips checkpoints, telemetry, and state transitions. | The observed double-SIGINT capture ends with `exit=0`: graceful SIGINT shutdown completed before the second signal's `forceShutdown` escalation ran. Treat that as a race limit of this capture, not as proof that `forceShutdown` completed or as an unconditional second-signal exit of `1`. For an interrupted shutdown, inspect the checkpoint commit's first line: it uses the `exit: [run:…] ` prefix. The checkpoint-created state/event field maps `exit` to `checkpointType: "skipped"`, because only `rig-setup`, `completed`, and `error` receive their same-named types. The Git commit prefix, the `checkpointType` field, and the process exit code are three distinct signals; do not use one as a substitute for another. Since 0.7.3, the shutdown watchdog arms on every `shutdown()` call that exits the process, including second-signal escalation to `forceShutdown`. The watchdog backstop is `SHUTDOWN_WATCHDOG_MS: 30000`; the cleanup delay after a normal exit is 100 ms. `outputFiles` can run `beforeCopy` commands before copying. A failure in either the before-copy operation or the copy operation fails the run. > **VersionNote:** Since 0.9.0, an `outputFiles` before-copy or copy failure produces exit code `1`. Before 0.9.0, that failure did not make the process exit with failure, so CI could miss a failure confined to copying output. `onFailure: "abort"` selects abort behavior. In `--headless` mode, since 0.7.3, a retriable failure fails fast with exit code `1` instead of waiting for an interactive client to initiate a retry. Budget metering is usage-time, not a prepaid guarantee. Each assistant message is priced and added to the running total; when `BudgetTracker.addCost` reaches `maxDollars`, it emits `exceeded` and the codon runner requests `SIGTERM`, so an already-billable message may push usage to or past the limit. The interrupt and `onExceeded` policy are separate: `onExceeded: "complete"` is the default and finalizes the interrupted codon as completed with partial output (run exit `0`); `"fail"` finalizes it as failed (exit `1`) and synthesizes `failureReason` `{type: "unknown", retriable: false, message: "Budget exceeded: …"}`. A `codon.completed` event may also carry `budgetExceeded: {currency, limit, used}`. Budget exhaustion is not a `loop.iteration.completed.terminationReason`; that field allows only `iteration_limit`, `context_exceeded`, `sentinel_skip`, or `failure`. Budget exhaustion is never classified via `classifyApiErrorText`. ## Codon-level exit information The process-level contract above is not the only place exit information appears. A codon's completion event and its state record expose different fields, and the shapes differ between them. | Location | Field | Shape | | ----------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | | `codon.completed` event | `exitStatus` | An object with `type: "success" \| "error" \| "killed"`, optional numeric `code`, and optional string `signal`. | | `codon.completed` event | `failureReason` | Optional `failureReason` object. | | Codon state record | `exitCode` | Plain number. | When a failed codon has no result and no crash text, the runtime synthesizes a failure reason. It is retriable when `sessionEstablished !== false`; when that setting is unknown, it defaults to true. Synthesis returns `undefined` for force-stop and `context-exceeded`. `context-exceeded` means that the model's context window was exceeded. `sentinel-load-failure` is constructed directly for a sentinel failure when `failCodonIfNotLoaded=true`, with `retriable: false` and `sentinelRefs`. `failCodonIfNotLoaded` is a codon-level setting that defaults to `false`. The failed codon's state record carries the recorded per-codon process exit code as a plain number, such as `1`, `-1`, or `130`; it is not the run-exit override. Only the optional `exitCode` parameter of `shutdown()`/`computeExitCode` overrides the computed exit contract.