Hankweave - Observe and debug a hank without guessing

A hank is a workflow made of codons (units of work). While a codon runs, the headless console (a run without an interactive display) can go quiet for minutes at a time, and silence tells you nothing about whether the run is healthy, stuck, or already finished. You do not have to guess. Every hankweave run leaves a trail of files in its execution directory: a runtime lock with a heartbeat, an event journal, a state file, agent logs, and a checkpoint repository. This page shows you how to read each of them, how to trace a failure from symptom to cause, and how to replay a finished run without spending API calls.

Throughout the page, <exec> stands for the execution directory, and the commands assume you run them from inside it.

Watch a live run without smothering it#

Check a running hank about every five minutes rather than every second. On a first run, we are learning its pace as well as checking its progress. The files to watch are the ones named by the runbook's observe step: events.jsonl, server.log, state.json, and the agentRoot/ outputs. Three checks cover most of what you need.

Inspect .hankweave/runtime.lock. Its pid identifies the process and lastHeartbeat is rewritten every 30 seconds; the runtime treats a heartbeat older than 120 seconds as stale.

cat .hankweave/runtime.lock

Check process liveness separately. If you launched hankweave through the package bin, its command line contains bin/hankweave; match it with pgrep -f bin/hankweave. A direct node dist/index.js launch will not match this pattern:

pgrep -f bin/hankweave

For a clean finish, find one codon.completed journal event carrying cost for each codon. Then check server.log for Shutting down server: all codons completed, State transition: RunCompleted, and the exit-0 shutdown result.

Check-it: compare the journal with the captured completion events below, then confirm the corresponding server.log shutdown lines. A completion event is lifecycle evidence; inspect the expected artifact on disk before calling the run successful.

Captured completion events. This capture shows the full event sequence for a single-codon run: RunStarted, the codon's state transitions from preparing through running, the alternating cost and message-count updates while the agent works, the final cost and checkpoint, and RunCompleted. Use it as a reference shape when reading your own journal.

{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunStarted", "runId": "<id>", "transition": {"type": "RunStarted", "data": {"runId": "<id>", "runFolder": "~/.hankweave-executions/<exec-id>/.hankweave/runs/<id>", "gitBranch": "run-<id>", "startingConditions": {"type": "fresh"}, "serverPid": 50024}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonStarted", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonStarted", "data": {"runId": "<id>", "codonId": "summarize-notes"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "summarize-notes", "from": "preparing", "to": "starting", "metadata": {}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "summarize-notes", "from": "starting", "to": "initializing", "metadata": {"claudePid": 962109, "claudeLogPath": ".hankweave/runs/<id>/summarize-notes-claude.log"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "summarize-notes", "from": "initializing", "to": "running", "metadata": {"claudeSessionId": "be7a1143-f289-4e61-9346-9c3bf0dc8ed2"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonFinalCostSet", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonFinalCostSet", "data": {"runId": "<id>", "codonId": "summarize-notes", "finalCost":"<n>", "finalTokens": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CheckpointCreated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CheckpointCreated", "data": {"runId": "<id>", "codonId": "summarize-notes", "checkpointType": "completed", "sha": "<sha>", "branch": "run-<id>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "summarize-notes", "from": "running", "to": "completed", "metadata": {"exitCode": 0, "resultMessageReceived": true, "checkpointSha": "<sha>", "contextExceeded": false, "extensionCount": 0}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "codon.completed", "data": {"codonId": "summarize-notes", "success": true, "cost":"<n>", "duration":"<n>", "exitStatus": {"type": "success"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunCompleted", "runId": "<id>", "transition": {"type": "RunCompleted", "data": {"runId": "<id>"}}, "resultingState": {"currentRunId": null, "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}

A second completion capture. The same lifecycle for a different codon (normalize-aster), so you can see which fields stay constant and which vary between runs.

{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunStarted", "runId": "<id>", "transition": {"type": "RunStarted", "data": {"runId": "<id>", "runFolder": "~/.hankweave-executions/<exec-id>/.hankweave/runs/<id>", "gitBranch": "run-<id>", "startingConditions": {"type": "fresh"}, "serverPid": 86887}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonStarted", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonStarted", "data": {"runId": "<id>", "codonId": "normalize-aster"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "normalize-aster", "from": "preparing", "to": "starting", "metadata": {}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "normalize-aster", "from": "starting", "to": "initializing", "metadata": {"claudePid": 948485, "claudeLogPath": ".hankweave/runs/<id>/normalize-aster-claude.log"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "normalize-aster", "from": "initializing", "to": "running", "metadata": {"claudeSessionId": "fa26d9e3-ac8b-4d27-976c-de60af1894e5"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonFinalCostSet", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonFinalCostSet", "data": {"runId": "<id>", "codonId": "normalize-aster", "finalCost":"<n>", "finalTokens": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CheckpointCreated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CheckpointCreated", "data": {"runId": "<id>", "codonId": "normalize-aster", "checkpointType": "completed", "sha": "<sha>", "branch": "run-<id>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "normalize-aster", "from": "running", "to": "completed", "metadata": {"exitCode": 0, "resultMessageReceived": true, "checkpointSha": "<sha>", "contextExceeded": false, "extensionCount": 0}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "codon.completed", "data": {"codonId": "normalize-aster", "success": true, "cost":"<n>", "duration":"<n>", "exitStatus": {"type": "success"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunCompleted", "runId": "<id>", "transition": {"type": "RunCompleted", "data": {"runId": "<id>"}}, "resultingState": {"currentRunId": null, "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}

Find why a codon failed in four stops#

When a codon fails, the evidence is spread across four places. From <exec>, follow the failure through them in order:

Open .hankweave/state.json, not a file at the execution root. Hankweave writes state.json.bak before each save.
Read the failed codon's exact claudeLogPath; never infer an agent-log filename.
Inspect .hankweave/events/events.jsonl.
Compare the sealed checkpoint (saved file state) with the checkpoint repository.

The state file is where the failure is recorded. A failed codon record contains failedDuring, exitCode, failureReason, partialCost, partialTokens, and, optionally, errorCheckpoint. The complete state shape is documented in reference/state-file. The stored failure-reason shape is shown here; use reference/errors-and-exit-codes for its first-match-wins classification taxonomy.

TS
// Failure reason schema
const failureReasonSchema = z.object({
  type: z.enum(["timeout", "rate-limit", "api-error", "sentinel-load-failure", "unknown"]),
  retriable: z.boolean(),
  message: z.string().optional(),
  sentinelRefs: z.array(z.string()).optional(), // Which sentinels failed (for sentinel-load-failure)
  /**
   * How long the provider asked us to wait before retrying, in milliseconds,
   * when its error carried an explicit hint (a Retry-After header value, a
   * "retry after 30s" phrasing, etc). Parsed by classifyApiErrorText and
   * consumed by computeRetryDelayMs, which prefers it over computed backoff —
   * the provider knows its own limit window better than we can guess.
   */
  retryAfterMs: z.number().optional(),
});

Read the codon's exit code without treating it as the whole diagnosis. The common values:

Scroll to explore the table →

Code	Meaning
`1`	General error
`-1`	Killed by a signal
`130`	Ctrl+C

New runs are at .runs[0] because the runtime unshifts them. To avoid relying on insertion order, sort by startTime and take the last run. These commands select the failed codon and total spend, then show the order-safe run selection:

⌁ Terminal
jq '.runs[0].codons[] | select(.status=="failed") | {codonId, failedDuring, failureReason, exitCode}' .hankweave/state.json
jq '[.runs[0].codons[] | (.finalCost // .partialCost // 0)] | add' .hankweave/state.json
jq '(.runs | sort_by(.startTime) | last)' .hankweave/state.json

A SIGTERM snapshot is different from a self-failing codon: immediately after the kill, the run and interrupted codon can still have status "running", with the codon's exitCode set to null and no failureReason. On resume, the run is marked "crashed" and that codon becomes "failed" with exitCode: -1; the kill itself does not capture the agent log.

Read the agent log without guessing its name#

Step 2 above sends you to the agent log; here is how to open the right one. Resolve the codon's claudeLogPath relative to <exec>. A stored path can look like .hankweave/runs/<runId>/<codonId>-claude.log. The run directory is created at run start, and <runId> has the form {unixMs}-{5-char random}; it has no run- prefix. That prefix belongs to checkpoint branches.

For both harnesses, replace # with - in an in-loop codon ID and use the -claude.log suffix. Thus normalize-next#0 maps to normalize-next-0-claude.log, but the value in state remains authoritative.

Read the file as JSONL (one JSON object per line) and use its last result line for the outcome. If a session dies before the SDK emits a result, the runtime appends a synthetic result with type:"result", subtype:"error", and is_error:true. An SDK timeout marker is different: a synthetic assistant message has message.model:"<synthetic>" and content API Error: Request timed out.; the runtime's timeout and context checks use that model marker.

Diff the changes the run actually made#

The state file tells you what the run reported; the checkpoint store tells you what it wrote. To compare what changed, select the checkpoint store explicitly rather than the main project's .git. The store is a non-bare repository with git directory <exec>/.hankweave/checkpoints/.hankweavecheckpoints and work tree agentRoot/. See Checkpoints for the mechanics. From the execution directory:

git --git-dir=.hankweave/checkpoints/.hankweavecheckpoints log --oneline
git --git-dir=.hankweave/checkpoints/.hankweavecheckpoints branch -a
git --git-dir=.hankweave/checkpoints/.hankweavecheckpoints diff run-<A> run-<B>

Checkpoint subjects begin {status}:{codonId} [run:{runId}] {codonName}. The rig (workspace setup stage) has status rig-setup, completed, error, or skipped. An interrupted shutdown can produce an exit: subject whose checkpoint type is skipped. The commit body carries Codon, Status, Timestamp, and Duration, and checkpoint commits are authored by froggie@southbridge.ai.

Only files matching the configured checkpointedFiles globs (file-pattern globs), after gitignore resolution, are committed from agentRoot/. A file outside those globs is invisible to rollback. If resume-after-failure rollback reports Target checkpoint a2f1b13… not found in manifest and removes workspace files, recover them from the error:package-gate checkpoint.

Classify failures by their phase#

Once you have the state record and the log, the fastest way to narrow the cause is failedDuring: preparing, starting, initializing, running, or completing-sentinels. Each phase points at a different suspect.

preparing: diagnose rig setup. A rig is the workspace setup stage, and its rig.setup.failed event carries failureType, optional exitCode and commandIndex, and ignored. If a command rig needs bun rigs/preflight.ts, deliver it with a copy rig first; otherwise the command rig can report ENOENT.
starting or initializing: inspect harness launch and self-test failures. The missing-key case is classified as auth and reports No authentication found (set ANTHROPIC_API_KEY).

Captured startup failure. This capture shows the self-test failing before any codon runs: the authentication check reports the missing ANTHROPIC_API_KEY, the server refuses to start, and the process exits 1.


╭────────────────────────────────────────────────────────────────────╮
│  Hankweave v0.10.0                                                 │
│  darwin arm64 • node v23.8.0                                       │
╰────────────────────────────────────────────────────────────────────╯

Created new execution directory: ~/.hankweave-executions/<exec-id>
New execution: <exec-id>
  Source → data
  Exec   → ~/.hankweave-executions/<exec-id>
  SDKs   → Claude node_modules ✓

[<ts>] [ERROR] Self-test completed: FAILED
[<ts>] [ERROR] Self-test FAILED: Some checks failed
[<ts>] [ERROR]   - authentication: ✗ No authentication found (set ANTHROPIC_API_KEY)
[ERROR] Server startup failed!
Error message: Self-test failed for 1 model(s):
  - Claude Haiku 4.5 (latest) (anthropic/claude-haiku-4-5): Some checks failed
      • authentication: No authentication found (set ANTHROPIC_API_KEY)
Stack trace:
Error: Self-test failed for 1 model(s):
  - Claude Haiku 4.5 (latest) (anthropic/claude-haiku-4-5): Some checks failed
      • authentication: No authentication found (set ANTHROPIC_API_KEY)
    at eU (file://<workspace>/build/runtime/0.10.0/node_modules/hankweave/dist/index.js:260:5401)
    at async Egt (file://<workspace>/build/runtime/0.10.0/node_modules/hankweave/dist/index.js:842:647)
    at async file://<workspace>/build/runtime/0.10.0/node_modules/hankweave/dist/index.js:843:754

exit=1

running: treat network, timeout, 429, and 5xx/overloaded failures as retriable; auth, billing/credit, and quota failures are permanent. A lived 429 with code 1308 was classified as non-retriable, so inspect the stored classification before retrying. A budget configuration without maxDollars or maxTimeSeconds leaves that failure unanticipated. With compaction (context trimming) off by default at 0.10.0, a plain codon can hit the context boundary; inspect the log for a Prompt is too long-class error or an output-token-maximum synthetic message.
completing-sentinels: the agent finished but checkpoint creation failed.

The three events below are the ones you will meet most often when classifying by phase: the codon completion record, the rig failure record, and the sentinel unload record.

Scroll to explore the table →

id	category	journaled	sentinelRouted	payloadFields	receipts
codon.completed	server-state	true	true	codonId, success, cost, duration, exitStatus, failureReason?, failureIgnored?, budgetExceeded?	schemas/event-schemas.ts:560, schemas/event-schemas.ts:961, schemas/event-schemas.ts:1235, hankweave-runtime.ts:1886, hankweave-runtime.ts:2064, hankweave-runtime.ts:2253, hankweave-runtime.ts:3365
rig.setup.failed	agentic-backbone	true	true	codonId, failureType, exitCode?, commandIndex?, ignored	schemas/event-schemas.ts:602, schemas/event-schemas.ts:991, schemas/event-schemas.ts:1243, hankweave-runtime.ts:1791
sentinel.unloaded	sentinel	true	false	sentinelId, codonId, reason, errorType?, finalCost, llmCallCount	schemas/event-schemas.ts:704, schemas/event-schemas.ts:1000, schemas/event-schemas.ts:1263, sentinels/sentinel-manager.ts:851

When a sentinel unloads, inspect its reason, finalCost, and llmCallCount. consecutive-failures means the sentinel's own calls kept failing, so check its key and quota before checking the hank. At 0.10.0, the sentinel manager can report No LLM provider available even while codons resolve the same key; treat that as a sentinel-side key-resolution symptom. A required sentinel (failCodonIfNotLoaded: true) that cannot load fails the codon while it is still starting, with failureReason type sentinel-load-failure, retriable:false, and sentinelRefs naming the sentinels.

Question signals that lie at 0.10.0#

Some signals look conclusive and are not. Before you trust any of them, check these known cases.

A success signal can be incomplete. An observed codon ended in success after about 15,000 output tokens of thinking, wrote no file, and sealed an empty checkpoint. Verify the artifact on disk rather than trusting the success event.

Budget-cap behavior belongs to the runbook: late maxTimeSeconds enforcement and onExceeded:"fail" can leave a post-codon.completed error: checkpoint with no outputs. An exhaustWithPrompt extension can likewise fail the run after its codon sealed; look for a completed: checkpoint alongside failed run state. The line aborted by user can describe a budget abort rather than a person stopping the run.

Keep process exit status separate from the codon's state exitCode. Graceful SIGINT, SIGTERM, or client-request shutdown normally computes process exit 0; forceShutdown schedules exit 1. The observed double-Ctrl+C capture ended at 0 because graceful shutdown won the race, so process exit 0 alone does not prove clean completion.

A first-party DeepSeek V4 Pro codon can reason silently for more than 15 minutes without streamed agent events. On a long judgment codon, the captured anchor run used --shim-idle-timeout 1800.

Re-watch a completed run with `--replay`#

Sometimes the question is not why a run failed but what it did, and you want to see the event stream again without paying for another execution. Replay covers that case: use it to inspect a completed execution's recorded event stream, develop a client against real events, or demo a run. Replay copies the execution directory to a temporary location, removes the copied runtime lock, discovers the hank and data from execution-meta.json, and paces output from recorded timestamps. It starts a fresh replay run and skips rigs, sentinels, and self-tests; normal replay makes no new provider calls, but it does not test changed rigs or external side effects. The temporary copy is deleted when the process exits, and the original execution remains intact.

Run it without assuming a global installation:

bunx hankweave@0.10.0 --replay <dir>

--replay cannot be combined with --execution, and it is one of the five flags omitted from help. The captured completed-execution replay shows the temporary copy, a fresh hankweave-replay-… run, and exit 0.

Replay capture


╭────────────────────────────────────────────────────────────────────╮
│  Hankweave v0.10.0                                                 │
│  darwin arm64 • node v23.8.0                                       │
╰────────────────────────────────────────────────────────────────────╯

[REPLAY] Using hank config from execution metadata: <workspace>/fixtures/scenarios/silent-reuse/hank.json
[REPLAY] Using data source from execution metadata: <workspace>/fixtures/scenarios/silent-reuse/data
[REPLAY] Copied execution dir to <capture-attempt><tmp>/hankweave-replay-1788702367070-4s0v
Resuming: hankweave-replay-1788702367070-4s0v
  Source → data
  Exec   → <capture-attempt><tmp>/hankweave-replay-1788702367070-4s0v
  SDKs   → Claude node_modules ✓

╭──────────────────────────────────────────────────────────────────────────────╮
│  Silent reuse v1.0.0                                                         │
│  1 codon • 0 loops                                                           │
╰──────────────────────────────────────────────────────────────────────────────╯

└─ [1] write-line (Write one line)
      model: haiku │ mode: fresh │ prompts: 1 (2 lines)
      checkpointedGlobs: 1


══════════════════════════════════════════════════
  Hankweave Server Started
  WebSocket: ws://localhost:<port>
══════════════════════════════════════════════════

Running in headless mode on port <port>
➜ Listening on: http://localhost:<port>/ (all interfaces)

exit=0

Avoid patterns that make debugging harder#

The techniques above work best when the response to a failure is deliberate. Before changing the hank, make the uncertainty explicit, and steer around these habits:

Infinite retry: read failureReason and its retriable classification before retrying an unchanged failure.
Prompt patching: identify why output went wrong before adding instructions.
Verification blindness: check the artifact on disk instead of accepting a success signal.
Debugging by deletion: do not remove codons to make a complex hank appear to work. --start-at and --end-at are not hankweave flags; use codon isolation, the extraction technique owned by testing and hardening.
Kitchen-sink prompt: do not bury needed context in an everything-at-once prompt.
Ignoring partial progress: keep a useful checkpoint while repairing a later failure.
Monolithic codon: split stages when you need to locate failures between them.

Once we have found the cause, use Resume, rollback, and retry for recovery procedures. For failure classes and exit contracts, see Errors and exit codes; for the state schema, see State file.

Watch a live run without smothering it#

Find why a codon failed in four stops#

Read the agent log without guessing its name#

Diff the changes the run actually made#

Classify failures by their phase#

Question signals that lie at 0.10.0#

Re-watch a completed run with --replay#

Avoid patterns that make debugging harder#

Re-watch a completed run with `--replay`#