# Observe and debug a hank without guessing

A hank is a workflow made of codons (units of work). While a codon runs, the headless console (a run without an interactive display) can go quiet for minutes at a time, and silence tells you nothing about whether the run is healthy, stuck, or already finished. You do not have to guess. Every hankweave run leaves a trail of files in its execution directory: a runtime lock with a heartbeat, an event journal, a state file, agent logs, and a checkpoint repository. This page shows you how to read each of them, how to trace a failure from symptom to cause, and how to replay a finished run without spending API calls.

Throughout the page, `<exec>` stands for the execution directory, and the commands assume you run them from inside it.

## Watch a live run without smothering it

Check a running hank about every five minutes rather than every second. On a first run, we are learning its pace as well as checking its progress. The files to watch are the ones named by the [runbook's observe step](/0.10.0/files/operate/runbook): `events.jsonl`, `server.log`, `state.json`, and the `agentRoot/` outputs. Three checks cover most of what you need.

1. Inspect `.hankweave/runtime.lock`. Its `pid` identifies the process and `lastHeartbeat` is rewritten every 30 seconds; the runtime treats a heartbeat older than 120 seconds as stale.

```bash
cat .hankweave/runtime.lock
```

2. Check process liveness separately. If you launched hankweave through the package bin, its command line contains `bin/hankweave`; match it with `pgrep -f bin/hankweave`. A direct `node dist/index.js` launch will not match this pattern:

```bash
pgrep -f bin/hankweave
```

3. For a clean finish, find one `codon.completed` journal event carrying `cost` for each codon. Then check `server.log` for `Shutting down server: all codons completed`, `State transition: RunCompleted`, and the exit-0 shutdown result.

**Check-it:** compare the journal with the captured completion events below, then confirm the corresponding `server.log` shutdown lines. A completion event is lifecycle evidence; inspect the expected artifact on disk before calling the run successful.

**Captured completion events.** This capture shows the full event sequence for a single-codon run: `RunStarted`, the codon's state transitions from `preparing` through `running`, the alternating cost and message-count updates while the agent works, the final cost and checkpoint, and `RunCompleted`. Use it as a reference shape when reading your own journal.

```text
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunStarted", "runId": "<id>", "transition": {"type": "RunStarted", "data": {"runId": "<id>", "runFolder": "~/.hankweave-executions/<exec-id>/.hankweave/runs/<id>", "gitBranch": "run-<id>", "startingConditions": {"type": "fresh"}, "serverPid": 50024}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonStarted", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonStarted", "data": {"runId": "<id>", "codonId": "summarize-notes"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "summarize-notes", "from": "preparing", "to": "starting", "metadata": {}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "summarize-notes", "from": "starting", "to": "initializing", "metadata": {"claudePid": 962109, "claudeLogPath": ".hankweave/runs/<id>/summarize-notes-claude.log"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "summarize-notes", "from": "initializing", "to": "running", "metadata": {"claudeSessionId": "be7a1143-f289-4e61-9346-9c3bf0dc8ed2"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "summarize-notes", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "summarize-notes", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonFinalCostSet", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonFinalCostSet", "data": {"runId": "<id>", "codonId": "summarize-notes", "finalCost":"<n>", "finalTokens": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CheckpointCreated", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CheckpointCreated", "data": {"runId": "<id>", "codonId": "summarize-notes", "checkpointType": "completed", "sha": "<sha>", "branch": "run-<id>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "summarize-notes", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "summarize-notes", "from": "running", "to": "completed", "metadata": {"exitCode": 0, "resultMessageReceived": true, "checkpointSha": "<sha>", "contextExceeded": false, "extensionCount": 0}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "codon.completed", "data": {"codonId": "summarize-notes", "success": true, "cost":"<n>", "duration":"<n>", "exitStatus": {"type": "success"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunCompleted", "runId": "<id>", "transition": {"type": "RunCompleted", "data": {"runId": "<id>"}}, "resultingState": {"currentRunId": null, "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
```

**A second completion capture.** The same lifecycle for a different codon (`normalize-aster`), so you can see which fields stay constant and which vary between runs.

```text
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunStarted", "runId": "<id>", "transition": {"type": "RunStarted", "data": {"runId": "<id>", "runFolder": "~/.hankweave-executions/<exec-id>/.hankweave/runs/<id>", "gitBranch": "run-<id>", "startingConditions": {"type": "fresh"}, "serverPid": 86887}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonStarted", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonStarted", "data": {"runId": "<id>", "codonId": "normalize-aster"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "normalize-aster", "from": "preparing", "to": "starting", "metadata": {}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "normalize-aster", "from": "starting", "to": "initializing", "metadata": {"claudePid": 948485, "claudeLogPath": ".hankweave/runs/<id>/normalize-aster-claude.log"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "normalize-aster", "from": "initializing", "to": "running", "metadata": {"claudeSessionId": "fa26d9e3-ac8b-4d27-976c-de60af1894e5"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CostsIncremented", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CostsIncremented", "data": {"runId": "<id>", "codonId": "normalize-aster", "costDelta":"<n>", "tokensDelta": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "AssistantMessageCountUpdated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "AssistantMessageCountUpdated", "data": {"runId": "<id>", "codonId": "normalize-aster", "newCount":"<n>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonFinalCostSet", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonFinalCostSet", "data": {"runId": "<id>", "codonId": "normalize-aster", "finalCost":"<n>", "finalTokens": {"inputTokens":"<n>", "outputTokens":"<n>", "cacheCreationTokens":"<n>", "cacheReadTokens":"<n>"}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CheckpointCreated", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CheckpointCreated", "data": {"runId": "<id>", "codonId": "normalize-aster", "checkpointType": "completed", "sha": "<sha>", "branch": "run-<id>"}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "CodonTransitioned", "runId": "<id>", "codonId": "normalize-aster", "transition": {"type": "CodonTransitioned", "data": {"runId": "<id>", "codonId": "normalize-aster", "from": "running", "to": "completed", "metadata": {"exitCode": 0, "resultMessageReceived": true, "checkpointSha": "<sha>", "contextExceeded": false, "extensionCount": 0}}}, "resultingState": {"currentRunId": "<id>", "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "codon.completed", "data": {"codonId": "normalize-aster", "success": true, "cost":"<n>", "duration":"<n>", "exitStatus": {"type": "success"}}}
{"id": "<id>", "timestamp": "<ts>", "type": "state.transition", "data": {"transitionType": "RunCompleted", "runId": "<id>", "transition": {"type": "RunCompleted", "data": {"runId": "<id>"}}, "resultingState": {"currentRunId": null, "runCount": 1, "totalCost":"<n>", "currentRunCost":"<n>"}}}
```

## Find why a codon failed in four stops

When a codon fails, the evidence is spread across four places. From `<exec>`, follow the failure through them in order:

1. Open `.hankweave/state.json`, not a file at the execution root. Hankweave writes `state.json.bak` before each save.
2. Read the failed codon's exact `claudeLogPath`; never infer an agent-log filename.
3. Inspect `.hankweave/events/events.jsonl`.
4. Compare the sealed checkpoint (saved file state) with the checkpoint repository.

The state file is where the failure is recorded. A failed codon record contains `failedDuring`, `exitCode`, `failureReason`, `partialCost`, `partialTokens`, and, optionally, `errorCheckpoint`. The complete state shape is documented in [reference/state-file](/0.10.0/files/reference/state-file). The stored failure-reason shape is shown here; use [reference/errors-and-exit-codes](/0.10.0/files/reference/errors-and-exit-codes) for its first-match-wins classification taxonomy.

```ts
// Failure reason schema
const failureReasonSchema = z.object({
  type: z.enum(["timeout", "rate-limit", "api-error", "sentinel-load-failure", "unknown"]),
  retriable: z.boolean(),
  message: z.string().optional(),
  sentinelRefs: z.array(z.string()).optional(), // Which sentinels failed (for sentinel-load-failure)
  /**
   * How long the provider asked us to wait before retrying, in milliseconds,
   * when its error carried an explicit hint (a Retry-After header value, a
   * "retry after 30s" phrasing, etc). Parsed by classifyApiErrorText and
   * consumed by computeRetryDelayMs, which prefers it over computed backoff —
   * the provider knows its own limit window better than we can guess.
   */
  retryAfterMs: z.number().optional(),
});
```

<a id="read-the-exit-code-without-over-interpreting-it"></a>
Read the codon's exit code without treating it as the whole diagnosis. The common values:

| Code  | Meaning            |
| ----- | ------------------ |
| `1`   | General error      |
| `-1`  | Killed by a signal |
| `130` | Ctrl+C             |

New runs are at `.runs[0]` because the runtime unshifts them. To avoid relying on insertion order, sort by `startTime` and take the last run. These commands select the failed codon and total spend, then show the order-safe run selection:

```bash
jq '.runs[0].codons[] | select(.status=="failed") | {codonId, failedDuring, failureReason, exitCode}' .hankweave/state.json
jq '[.runs[0].codons[] | (.finalCost // .partialCost // 0)] | add' .hankweave/state.json
jq '(.runs | sort_by(.startTime) | last)' .hankweave/state.json
```

A SIGTERM snapshot is different from a self-failing codon: immediately after the kill, the run and interrupted codon can still have status `"running"`, with the codon's `exitCode` set to `null` and no `failureReason`. On resume, the run is marked `"crashed"` and that codon becomes `"failed"` with `exitCode: -1`; the kill itself does not capture the agent log.

## Read the agent log without guessing its name

Step 2 above sends you to the agent log; here is how to open the right one. Resolve the codon's `claudeLogPath` relative to `<exec>`. A stored path can look like `.hankweave/runs/<runId>/<codonId>-claude.log`. The run directory is created at run start, and `<runId>` has the form `{unixMs}-{5-char random}`; it has no `run-` prefix. That prefix belongs to checkpoint branches.

For both harnesses, replace `#` with `-` in an in-loop codon ID and use the `-claude.log` suffix. Thus `normalize-next#0` maps to `normalize-next-0-claude.log`, but the value in state remains authoritative.

Read the file as JSONL (one JSON object per line) and use its last `result` line for the outcome. If a session dies before the SDK emits a result, the runtime appends a synthetic result with `type:"result"`, `subtype:"error"`, and `is_error:true`. An SDK timeout marker is different: a synthetic assistant message has `message.model:"<synthetic>"` and content `API Error: Request timed out.`; the runtime's timeout and context checks use that model marker.

> **Pitfall:** Do not reconstruct an agent-log name from a codon ID or model. Read `claudeLogPath` from state, then open that path under `.hankweave/runs/<runId>/`.

## Diff the changes the run actually made

The state file tells you what the run reported; the checkpoint store tells you what it wrote. To compare what changed, select the checkpoint store explicitly rather than the main project's `.git`. The store is a non-bare repository with git directory `<exec>/.hankweave/checkpoints/.hankweavecheckpoints` and work tree `agentRoot/`. See [Checkpoints](/0.10.0/files/concepts/checkpoints) for the mechanics. From the execution directory:

```bash
git --git-dir=.hankweave/checkpoints/.hankweavecheckpoints log --oneline
git --git-dir=.hankweave/checkpoints/.hankweavecheckpoints branch -a
git --git-dir=.hankweave/checkpoints/.hankweavecheckpoints diff run-<A> run-<B>
```

Checkpoint subjects begin `{status}:{codonId} [run:{runId}] {codonName}`. The `rig` (workspace setup stage) has status `rig-setup`, `completed`, `error`, or `skipped`. An interrupted shutdown can produce an `exit:` subject whose checkpoint type is `skipped`. The commit body carries Codon, Status, Timestamp, and Duration, and checkpoint commits are authored by `froggie@southbridge.ai`.

Only files matching the configured `checkpointedFiles` globs (file-pattern globs), after gitignore resolution, are committed from `agentRoot/`. A file outside those globs is invisible to rollback. If resume-after-failure rollback reports `Target checkpoint a2f1b13… not found in manifest` and removes workspace files, recover them from the `error:package-gate` checkpoint.

## Classify failures by their phase

Once you have the state record and the log, the fastest way to narrow the cause is `failedDuring`: `preparing`, `starting`, `initializing`, `running`, or `completing-sentinels`. Each phase points at a different suspect.

* **`preparing`:** diagnose rig setup. A rig is the workspace setup stage, and its `rig.setup.failed` event carries `failureType`, optional `exitCode` and `commandIndex`, and `ignored`. If a command rig needs `bun rigs/preflight.ts`, deliver it with a copy rig first; otherwise the command rig can report `ENOENT`.
* **`starting` or `initializing`:** inspect harness launch and self-test failures. The missing-key case is classified as `auth` and reports `No authentication found (set ANTHROPIC_API_KEY)`.

**Captured startup failure.** This capture shows the self-test failing before any codon runs: the authentication check reports the missing `ANTHROPIC_API_KEY`, the server refuses to start, and the process exits 1.

```text

╭────────────────────────────────────────────────────────────────────╮
│  Hankweave v0.10.0                                                 │
│  darwin arm64 • node v23.8.0                                       │
╰────────────────────────────────────────────────────────────────────╯

Created new execution directory: ~/.hankweave-executions/<exec-id>
New execution: <exec-id>
  Source → data
  Exec   → ~/.hankweave-executions/<exec-id>
  SDKs   → Claude node_modules ✓

[<ts>] [ERROR] Self-test completed: FAILED
[<ts>] [ERROR] Self-test FAILED: Some checks failed
[<ts>] [ERROR]   - authentication: ✗ No authentication found (set ANTHROPIC_API_KEY)
[ERROR] Server startup failed!
Error message: Self-test failed for 1 model(s):
  - Claude Haiku 4.5 (latest) (anthropic/claude-haiku-4-5): Some checks failed
      • authentication: No authentication found (set ANTHROPIC_API_KEY)
Stack trace:
Error: Self-test failed for 1 model(s):
  - Claude Haiku 4.5 (latest) (anthropic/claude-haiku-4-5): Some checks failed
      • authentication: No authentication found (set ANTHROPIC_API_KEY)
    at eU (file://<workspace>/build/runtime/0.10.0/node_modules/hankweave/dist/index.js:260:5401)
    at async Egt (file://<workspace>/build/runtime/0.10.0/node_modules/hankweave/dist/index.js:842:647)
    at async file://<workspace>/build/runtime/0.10.0/node_modules/hankweave/dist/index.js:843:754

exit=1
```

* **`running`:** treat network, timeout, 429, and 5xx/overloaded failures as retriable; auth, billing/credit, and quota failures are permanent. A lived 429 with code 1308 was classified as non-retriable, so inspect the stored classification before retrying. A budget configuration without `maxDollars` or `maxTimeSeconds` leaves that failure unanticipated. With compaction (context trimming) off by default at 0.10.0, a plain codon can hit the context boundary; inspect the log for a `Prompt is too long`-class error or an output-token-maximum synthetic message.
* **`completing-sentinels`:** the agent finished but checkpoint creation failed.

The three events below are the ones you will meet most often when classifying by phase: the codon completion record, the rig failure record, and the sentinel unload record.

| id                | category         | journaled | sentinelRouted | payloadFields                                                                                  | receipts                                                                                                                                                                                              |
| ----------------- | ---------------- | --------- | -------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| codon.completed   | server-state     | true      | true           | codonId, success, cost, duration, exitStatus, failureReason?, failureIgnored?, budgetExceeded? | schemas/event-schemas.ts:560, schemas/event-schemas.ts:961, schemas/event-schemas.ts:1235, hankweave-runtime.ts:1886, hankweave-runtime.ts:2064, hankweave-runtime.ts:2253, hankweave-runtime.ts:3365 |
| rig.setup.failed  | agentic-backbone | true      | true           | codonId, failureType, exitCode?, commandIndex?, ignored                                        | schemas/event-schemas.ts:602, schemas/event-schemas.ts:991, schemas/event-schemas.ts:1243, hankweave-runtime.ts:1791                                                                                  |
| sentinel.unloaded | sentinel         | true      | false          | sentinelId, codonId, reason, errorType?, finalCost, llmCallCount                               | schemas/event-schemas.ts:704, schemas/event-schemas.ts:1000, schemas/event-schemas.ts:1263, sentinels/sentinel-manager.ts:851                                                                         |

When a sentinel unloads, inspect its `reason`, `finalCost`, and `llmCallCount`. `consecutive-failures` means the sentinel's own calls kept failing, so check its key and quota before checking the hank. At 0.10.0, the sentinel manager can report `No LLM provider available` even while codons resolve the same key; treat that as a sentinel-side key-resolution symptom. A required sentinel (`failCodonIfNotLoaded: true`) that cannot load fails the codon while it is still starting, with `failureReason` type `sentinel-load-failure`, `retriable:false`, and `sentinelRefs` naming the sentinels.

> **VersionNote:** In 0.8.0, provider retry classification changed and compaction became off by default. Re-evaluate older failure-reading habits at 0.10.0.

## Question signals that lie at 0.10.0

Some signals look conclusive and are not. Before you trust any of them, check these known cases.

A success signal can be incomplete. An observed codon ended in `success` after about 15,000 output tokens of thinking, wrote no file, and sealed an empty checkpoint. Verify the artifact on disk rather than trusting the success event.

Budget-cap behavior belongs to the [runbook](/0.10.0/files/operate/runbook): late `maxTimeSeconds` enforcement and `onExceeded:"fail"` can leave a post-`codon.completed` `error:` checkpoint with no outputs. An `exhaustWithPrompt` extension can likewise fail the run after its codon sealed; look for a `completed:` checkpoint alongside failed run state. The line `aborted by user` can describe a budget abort rather than a person stopping the run.

Keep process exit status separate from the codon's state `exitCode`. Graceful SIGINT, SIGTERM, or client-request shutdown normally computes process exit 0; `forceShutdown` schedules exit 1. The observed double-Ctrl+C capture ended at 0 because graceful shutdown won the race, so process exit 0 alone does not prove clean completion.

A first-party DeepSeek V4 Pro codon can reason silently for more than 15 minutes without streamed agent events. On a long judgment codon, the captured anchor run used `--shim-idle-timeout 1800`.

<a id="re-watch-a-run-without-spending-api-calls"></a>

## Re-watch a completed run with `--replay`

Sometimes the question is not why a run failed but what it did, and you want to see the event stream again without paying for another execution. Replay covers that case: use it to inspect a completed execution's recorded event stream, develop a client against real events, or demo a run. Replay copies the execution directory to a temporary location, removes the copied runtime lock, discovers the hank and data from `execution-meta.json`, and paces output from recorded timestamps. It starts a fresh replay run and skips rigs, sentinels, and self-tests; normal replay makes no new provider calls, but it does not test changed rigs or external side effects. The temporary copy is deleted when the process exits, and the original execution remains intact.

Run it without assuming a global installation:

```bash
bunx hankweave@0.10.0 --replay <dir>
```

`--replay` cannot be combined with `--execution`, and it is one of the five flags omitted from help. The captured completed-execution replay shows the temporary copy, a fresh `hankweave-replay-…` run, and exit 0.

**Replay capture**

```text

╭────────────────────────────────────────────────────────────────────╮
│  Hankweave v0.10.0                                                 │
│  darwin arm64 • node v23.8.0                                       │
╰────────────────────────────────────────────────────────────────────╯

[REPLAY] Using hank config from execution metadata: <workspace>/fixtures/scenarios/silent-reuse/hank.json
[REPLAY] Using data source from execution metadata: <workspace>/fixtures/scenarios/silent-reuse/data
[REPLAY] Copied execution dir to <capture-attempt><tmp>/hankweave-replay-1788702367070-4s0v
Resuming: hankweave-replay-1788702367070-4s0v
  Source → data
  Exec   → <capture-attempt><tmp>/hankweave-replay-1788702367070-4s0v
  SDKs   → Claude node_modules ✓

╭──────────────────────────────────────────────────────────────────────────────╮
│  Silent reuse v1.0.0                                                         │
│  1 codon • 0 loops                                                           │
╰──────────────────────────────────────────────────────────────────────────────╯

└─ [1] write-line (Write one line)
      model: haiku │ mode: fresh │ prompts: 1 (2 lines)
      checkpointedGlobs: 1


══════════════════════════════════════════════════
  Hankweave Server Started
  WebSocket: ws://localhost:<port>
══════════════════════════════════════════════════

Running in headless mode on port <port>
➜ Listening on: http://localhost:<port>/ (all interfaces)

exit=0
```

## Avoid patterns that make debugging harder

The techniques above work best when the response to a failure is deliberate. Before changing the hank, make the uncertainty explicit, and steer around these habits:

* **Infinite retry:** read `failureReason` and its retriable classification before retrying an unchanged failure.
* **Prompt patching:** identify why output went wrong before adding instructions.
* **Verification blindness:** check the artifact on disk instead of accepting a success signal.
* **Debugging by deletion:** do not remove codons to make a complex hank appear to work. `--start-at` and `--end-at` are not hankweave flags; use codon isolation, the extraction technique owned by [testing and hardening](/0.10.0/files/author/testing-and-hardening).
* **Kitchen-sink prompt:** do not bury needed context in an everything-at-once prompt.
* **Ignoring partial progress:** keep a useful checkpoint while repairing a later failure.
* **Monolithic codon:** split stages when you need to locate failures between them.

Once we have found the cause, use [Resume, rollback, and retry](/0.10.0/files/operate/resume-rollback-and-retry) for recovery procedures. For failure classes and exit contracts, see [Errors and exit codes](/0.10.0/files/reference/errors-and-exit-codes); for the state schema, see [State file](/0.10.0/files/reference/state-file).