# Observe runs and traces When a run finishes – or stalls – the first question is what actually happened: which codons ran, what they cost, and where a failure occurred. Hankweave gives you two ways to answer that. Every run writes an append-only execution journal to disk, so you can inspect outcomes locally with nothing more than `jq`. When you want a richer view – a span tree with LLM calls, tool calls, and per-span costs – the standalone `hankweave-trace` package uploads the same execution directory to Braintrust or Langfuse. This page covers both, starting with the journal and then the two ways to get a trace uploaded: running `hankweave-trace` yourself after a run, or letting the runtime do it on shutdown. Neither path is related to Hankweave's own product telemetry, which the last section disentangles. ## Read a run without a platform The journal is the fastest way to inspect a run because it needs no account, no credentials, and no extra tooling. Each execution writes an append-only log at `/.hankweave/events/events.jsonl`, where `` is the execution directory, such as `$HOME/.hankweave-executions/`; run the shell commands below from that directory. Each line is one JSON object with the envelope `{id, timestamp, type, data}`, and the event-specific fields live inside `data`. One ordering caveat matters before you build anything on this file: the journal is written in emission order, not sorted by `timestamp`, so do not infer chronology by sorting or by assuming adjacent timestamps. The journal includes server-state, agentic-backbone, and sentinel events. Connection-state events such as `server.ready`, `pong`, `history.batch`, and `incomplete.codon` are not journaled. The catalog at [/reference/events](/0.10.0/files/reference/events) lists the event types and payload fields; [/integrate/event-journal](/0.10.0/files/integrate/event-journal) covers the journal and its storage choices. For adjacent views, see [/reference/state-file](/0.10.0/files/reference/state-file), [/integrate/client-and-exported-types](/0.10.0/files/integrate/client-and-exported-types), and [/operate/observe-and-debug](/0.10.0/files/operate/observe-and-debug). In lived 0.10.0 runs started in headless mode (without an interactive console), the console is silent during codons; monitor `agentRoot/` files and `.hankweave/events/events.jsonl` instead. The table below lists the journal events you will reach for most often when reconstructing a run, with their payload fields and the source locations that define them: | id | category | journaled | sentinelRouted | payloadFields | receipts | | ------------------------ | ------------ | --------- | -------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | budget.summary | server-state | true | true | ceiling, allocation, rows, totals | schemas/event-schemas.ts:724, schemas/event-schemas.ts:979, schemas/event-schemas.ts:1267, hankweave-runtime.ts:6478 | | codon.completed | server-state | true | true | codonId, success, cost, duration, exitStatus, failureReason?, failureIgnored?, budgetExceeded? | schemas/event-schemas.ts:560, schemas/event-schemas.ts:961, schemas/event-schemas.ts:1235, hankweave-runtime.ts:1886, hankweave-runtime.ts:2064, hankweave-runtime.ts:2253, hankweave-runtime.ts:3365 | | loop.iteration.completed | server-state | true | true | loopId, iteration, durationMs, costUsd, tokensUsed, isFinal, terminationReason? | schemas/event-schemas.ts:612, schemas/event-schemas.ts:976, schemas/event-schemas.ts:1245, hankweave-runtime.ts:6035 | | sentinel.output | sentinel | true | false | sentinelId, codonId, triggerNumber, outputType, content, cost, tokens, eventCount | schemas/event-schemas.ts:714, schemas/event-schemas.ts:1002, schemas/event-schemas.ts:1265, sentinels/sentinel.ts:538, sentinels/sentinel.ts:635, sentinels/sentinel.ts:762, sentinels/sentinel.ts:877 | | sentinel.unloaded | sentinel | true | false | sentinelId, codonId, reason, errorType?, finalCost, llmCallCount | schemas/event-schemas.ts:704, schemas/event-schemas.ts:1000, schemas/event-schemas.ts:1263, sentinels/sentinel-manager.ts:851 | | state.snapshot | server-state | true | true | currentCodon?, completedCodons, fileTree, totalCost, totalTime, recentFileAccess?, isRollingBack | schemas/event-schemas.ts:550, schemas/event-schemas.ts:963, schemas/event-schemas.ts:1233, hankweave-runtime.ts:1422 | | token.usage | server-state | true | true | codonId, inputTokens, outputTokens, cacheCreationTokens, cacheReadTokens, totalCost, modelId?, modelUsage? | schemas/event-schemas.ts:577, schemas/event-schemas.ts:965, schemas/event-schemas.ts:1238, hankweave-runtime.ts:2646, hankweave-runtime.ts:2660 | The cost-bearing fields have different scopes: `token.usage` carries input, output, cache-creation, cache-read, total-cost, and optional model fields; `codon.completed` carries success, cost, duration, exit status, and an optional failure reason; loop, sentinel, snapshot, and budget events carry their own totals. Treat `budget.summary` as the runtime's state-based summary, not as a provider invoice or a guaranteed retry-inclusive total. The captured journal line below is the raw envelope, with identifiers and values normalized. The complete capture also shows the surrounding `state.transition` records; it is not a claim that a run contains a fixed number of events. ```jsonl {"id": "", "timestamp": "", "type": "codon.completed", "data": {"codonId": "summarize-notes", "success": true, "cost":"", "duration":"", "exitStatus": {"type": "success"}}} ``` To watch terminal codon outcomes as they arrive, we'll stream JSONL records and select the event type: ```sh tail -f .hankweave/events/events.jsonl | jq -c 'select(.type == "codon.completed")' ``` After the run, we can inspect the reported cost at `.data.cost`: ```sh jq -c 'select(.type == "codon.completed") | {id, codonId: .data.codonId, cost: .data.cost, success: .data.success}' \ .hankweave/events/events.jsonl ``` Two summation traps follow from how these events are emitted. Do not sum every `token.usage` record: that event is emitted for incremental per-call usage and again for the codon's final total. Do not sum every `codon.completed` record when retries are present: there is one terminal event per attempt, and the final event's cost already includes accumulated retry cost. For a per-run reported-codon-cost view, establish the run boundaries from `state.transition`, deduplicate retransmitted event IDs, retain the final `codon.completed` for each `(runId, runtime codonId)`, and sum only that view. Keep continuation runs and loop runtime IDs distinct. This is a recipe for that defined scope, not a captured end-to-end aggregation test; interrupted or skipped work, extensions, sentinels, health checks, and unpriced routes need their own scope decisions. A complete live execution with one tracked codon recorded its cost separately from provider health checks and sentinel calls: ```text - Tracked codon cost: $0.10441000 (provider health checks and sentinel calls are separate) ``` **Check it:** after a run, confirm that `.hankweave/events/events.jsonl` exists and that a clean run has a `codon.completed` record for its terminal codon; a retry adds a record for each failed attempt. The normalized fixture shows the `.data.cost` path, and the complete execution capture records the tracked codon cost without claiming that it is the provider's invoice. > **Pitfall:** At 0.10.0, `pi/zai/*` codons emit `codon.completed` with cost `0`; JSONL-derived totals and trace dashboards therefore under-report zai spend, and `budget.maxDollars` does not trip for that spend. Use `maxTimeSeconds` as the guard. See [/operate/runbook](/0.10.0/files/operate/runbook) for budget outcomes. ## Upload a finished run to Braintrust or Langfuse The journal answers "what happened" as flat records. For a hierarchical view – codons containing LLM calls and tool calls, with costs rolled up – use the standalone `hankweave-trace` package to transform an execution directory containing the run's state, journal, and per-codon logs into a trace tree; the journal is stored at `/.hankweave/events/events.jsonl`. Its documented modes are `upload` for post-run batches, `watch` for live spans followed by a final upload, and `generate` for provider payload JSON on stdout with progress on stderr; `generate` does not require credentials. These package details are documented since 0.7.0, so verify them against the version you install. For a post-run upload, select a platform and pass the execution directory. Set one of the credential variables listed in [Control credentials, content, and re-uploads](#control-credentials-content-and-re-uploads) before running this command; do not put a real key in a page or shell history: ```sh EXEC_DIR="$HOME/.hankweave-executions/your-execution-id" npx -y hankweave-trace upload "$EXEC_DIR" --braintrust ``` The package surface documents `--braintrust` and `--langfuse` platform selection, `--project ` for the Braintrust project (default `Hankweave`), `--latest-only`, `--redact`, `--force`, and comma-separated `--tags`. Confirm the installed package's help before relying on a flag: this surface is not part of Hankweave's locked runtime artifact. Once uploaded, a trace has one root for a Hankweave run. Its child spans represent codons, loops, LLM calls, tool calls, rig setup, and sentinels; the root carries run-level cost, duration, status, and tags, while loops group iterations. The exact placement of tokens and costs differs by codon harness; see [Trust the cost numbers in a trace](#trust-the-cost-numbers-in-a-trace) before comparing spans. If credentials are missing, the package tells you so directly. The pinned `config` capture below shows how that is reported at 0.10.0: ```text hankweave-trace configuration ============================== Braintrust: not configured Issue: missing HANKWEAVE_TRACE_BRAINTRUST_API_KEY Langfuse: not configured Issue: missing HANKWEAVE_TRACE_LANGFUSE_PUBLIC_KEY and HANKWEAVE_TRACE_LANGFUSE_SECRET_KEY Config file: none found Issues: - No tracing platform configured. Set HANKWEAVE_TRACE_BRAINTRUST_API_KEY or HANKWEAVE_TRACE_LANGFUSE_* env vars. npm notice npm notice New major version of npm available! 10.9.2 -> 12.0.2 npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.0.2 npm notice To update run: npm install -g npm@12.0.2 npm notice exit=0 ``` **Check it:** for the standalone package, verify the installed version's `upload`, `watch`, and `generate` behavior rather than treating this page's historical package surface as a locked CLI contract. The captured `config` output confirms the package's configuration surface, not a successful upload. ## Let the runtime upload the trace for you Running `hankweave-trace` by hand works well for a finished run, but it is easy to forget. The runtime can invoke the package itself: at startup it checks the trace configuration, and on shutdown it uploads before exiting. Set `HANKWEAVE_TRACE_BRAINTRUST` and/or `HANKWEAVE_TRACE_LANGFUSE` as presence-based enable flags. They are distinct from the credential variables consumed by `hankweave-trace`. Under Bun the runtime defaults to `bunx hankweave-trace`; otherwise it uses `npx hankweave-trace`. Set `HANKWEAVE_TRACE_BINARY` when you need a pinned global install or an air-gapped executable. At startup, the runtime runs ` config` with a 30-second timeout and prints the configuration block. If the output contains configuration issues, it logs a warning before the run has spent time on codons. On graceful shutdown it synchronously runs the upload before exiting; force-shutdown gets a best-effort call, and an upload guard prevents a second upload. The runtime's upload command always includes `--force`, so its own re-uploads bypass the manual dedup marker; deterministic (derived consistently from run data) span IDs make that safe. A real 0.10.0 run with `HANKWEAVE_TRACE_BRAINTRUST=1` and no credential shows the startup configuration and the attempted shutdown upload. The attempt fails because the credential is absent; the capture does not demonstrate a successful platform upload: ```text --- hankweave-trace config --- hankweave-trace configuration ============================== Braintrust: not configured Issue: missing HANKWEAVE_TRACE_BRAINTRUST_API_KEY Langfuse: not configured Issue: missing HANKWEAVE_TRACE_LANGFUSE_PUBLIC_KEY and HANKWEAVE_TRACE_LANGFUSE_SECRET_KEY Config file: none found Issues: - No tracing platform configured. Set HANKWEAVE_TRACE_BRAINTRUST_API_KEY or HANKWEAVE_TRACE_LANGFUSE_* env vars. [hankweave-trace config stderr] npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7 [] [ERROR] [hankweave-trace config stderr] npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7 [hankweave-trace] WARNING: tracing is misconfigured — upload on exit will likely fail. [] [ERROR] [hankweave-trace] WARNING: tracing is misconfigured — upload on exit will likely fail. Running in headless mode on port ➜ Listening on: http://localhost:/ (all interfaces) > Uploading trace: npx hankweave-trace upload "~/.hankweave-executions/" --force --braintrust npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7 Error: --braintrust requires a Braintrust API key (env, config file, or HANKWEAVE_TRACE_BRAINTRUST_API_KEY) [] [ERROR] npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7 Error: --braintrust requires a Braintrust API key (env, config file, or HANKWEAVE_TRACE_BRAINTRUST_API_KEY) ``` **Check it:** with a tracing flag enabled, look for the `--- hankweave-trace config ---` block at startup and an upload command containing `--force --braintrust` at shutdown. The capture shows both, including the absent-credential failure. If an enabled platform lacks credentials, startup emits this warning in the server log: ```text 41:[] [INFO] --- hankweave-trace config --- 42:[] [INFO] hankweave-trace configuration 54:[] [INFO] - No tracing platform configured. Set HANKWEAVE_TRACE_BRAINTRUST_API_KEY or HANKWEAVE_TRACE_LANGFUSE_* env vars. 55:[] [ERROR] [hankweave-trace config stderr] npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7 56:[] [ERROR] [hankweave-trace] WARNING: tracing is misconfigured — upload on exit will likely fail. ``` **Check it:** treat the warning as a configuration failure to fix before relying on the trace, not as evidence that an upload succeeded. Runtime trace variables are documented with the rest of the environment variables at [/reference/environment-variables](/0.10.0/files/reference/environment-variables). ## Control credentials, content, and re-uploads Whichever upload path you use, the same package-level settings decide where the trace goes and what it contains. These `hankweave-trace` controls are documented since 0.7.0; verify them against the version you install before relying on a default or config path. Choose the credentials and content controls for your destination: * `HANKWEAVE_TRACE_BRAINTRUST_API_KEY` and `HANKWEAVE_TRACE_BRAINTRUST_PROJECT` (default `Hankweave`). * `HANKWEAVE_TRACE_LANGFUSE_PUBLIC_KEY`, `HANKWEAVE_TRACE_LANGFUSE_SECRET_KEY`, and `HANKWEAVE_TRACE_LANGFUSE_BASE_URL` (default `https://cloud.langfuse.com`). * `HANKWEAVE_TRACE_TAGS` and `HANKWEAVE_TRACE_REDACT`. It reads `.hankweave-trace.json` in the working directory, then `~/.config/hankweave-trace/config.json`. Values beginning with `$` resolve from the environment. Precedence is CLI flags, environment variables, config file, then defaults. Configure both platforms if you want both to receive the trace; a failure on one does not block the other. Use `--redact` or `HANKWEAVE_TRACE_REDACT=1` to replace prompts, model output, tool input and output, thinking blocks, and sentinel observations with `[redacted]`. Span structure, names, metrics, errors, status, tags, and model names remain. Span IDs are deterministic from run data and platforms upsert by ID. Manual re-upload is blocked by `.hankweave/tracing-marker.json` unless you pass `--force`. ## Trust the cost numbers in a trace Trace dashboards show token and cost figures, but where those figures sit – and how the platform computes them – depends on the codon harness and the destination. These `hankweave-trace` span-placement and cost-display behaviors are documented since 0.7.0; verify them against the version you install. Claude SDK codons put token counts on individual LLM-call spans, leaving parent spans without those tokens; both platforms sum from children, so summing parent and child tokens double-counts. Pi-harness codons do not expose a per-message breakdown, so their token counts sit on the codon span and child LLM spans show zero. Braintrust computes its own estimate from tokens, so the trace stores Hankweave's computed cost in `metadata.hankweaveCost`. Langfuse can overestimate Anthropic costs when prompt caching is involved; the trace overrides each generation's `totalCost` with the Hankweave-computed value. These are trace values, not a provider invoice. A trace built from the journal inherits the journal's cost fields, including the 0 cost reported by `pi/zai/*` codons at 0.10.0. ## Choose between Braintrust and Langfuse The error-visibility and platform-comparison behavior below is documented since 0.7.0; verify it against the version you install. For this 0.10.0 documentation snapshot, Braintrust is cloud-only and provides generic spans plus AI features such as chart builder and topic maps. Langfuse is open source and can be self-hosted or used as a cloud service; it provides a native `generation` type and sessions for grouping runs of the same hank (a named execution definition). Both platforms surface failed runs, failed codons and their failure reasons, tool errors, rig-setup failures, and sentinel error unloads on the relevant spans. Langfuse additionally marks the trace with `level: "ERROR"` when the run fails. Choose the platform based on hosting and grouping needs, then keep the span-cost caveats above in view. ## Tell traces apart from Hankweave telemetry Hankweave's product telemetry is a separate pipeline from `hankweave-trace`. It is enabled by default and sends to the self-hosted PostHog endpoint `https://hw-telemetry.southbridge.ai`; opt-out precedence is `DO_NOT_TRACK=1`, then `HANKWEAVE_TELEMETRY=0|false`, CI auto-detection, and finally `hankweave.json` with `telemetry.enabled=false`. The full contract is in [/reference/telemetry](/0.10.0/files/reference/telemetry). Your trace upload instead runs a separate binary against your execution directory and sends to the Braintrust or Langfuse endpoint selected by your credentials. This is a separate path from product telemetry. > **VersionNote:** Product telemetry follows the opt-out contract above, while `hankweave-trace` uses a separate binary and your selected Braintrust or Langfuse endpoint.