Observe runs and traces

When a run finishes – or stalls – the first question is what actually happened: which codons ran, what they cost, and where a failure occurred. Hankweave gives you two ways to answer that. Every run writes an append-only execution journal to disk, so you can inspect outcomes locally with nothing more than jq. When you want a richer view – a span tree with LLM calls, tool calls, and per-span costs – the standalone hankweave-trace package uploads the same execution directory to Braintrust or Langfuse. This page covers both, starting with the journal and then the two ways to get a trace uploaded: running hankweave-trace yourself after a run, or letting the runtime do it on shutdown. Neither path is related to Hankweave's own product telemetry, which the last section disentangles.

Read a run without a platform#

The journal is the fastest way to inspect a run because it needs no account, no credentials, and no extra tooling. Each execution writes an append-only log at <exec>/.hankweave/events/events.jsonl, where <exec> is the execution directory, such as $HOME/.hankweave-executions/<exec-id>; run the shell commands below from that directory. Each line is one JSON object with the envelope {id, timestamp, type, data}, and the event-specific fields live inside data. One ordering caveat matters before you build anything on this file: the journal is written in emission order, not sorted by timestamp, so do not infer chronology by sorting or by assuming adjacent timestamps.

The journal includes server-state, agentic-backbone, and sentinel events. Connection-state events such as server.ready, pong, history.batch, and incomplete.codon are not journaled. The catalog at /reference/events lists the event types and payload fields; /integrate/event-journal covers the journal and its storage choices. For adjacent views, see /reference/state-file, /integrate/client-and-exported-types, and /operate/observe-and-debug.

In lived 0.10.0 runs started in headless mode (without an interactive console), the console is silent during codons; monitor agentRoot/ files and .hankweave/events/events.jsonl instead.

The table below lists the journal events you will reach for most often when reconstructing a run, with their payload fields and the source locations that define them:

The cost-bearing fields have different scopes: token.usage carries input, output, cache-creation, cache-read, total-cost, and optional model fields; codon.completed carries success, cost, duration, exit status, and an optional failure reason; loop, sentinel, snapshot, and budget events carry their own totals. Treat budget.summary as the runtime's state-based summary, not as a provider invoice or a guaranteed retry-inclusive total.

The captured journal line below is the raw envelope, with identifiers and values normalized. The complete capture also shows the surrounding state.transition records; it is not a claim that a run contains a fixed number of events.

{"id": "<id>", "timestamp": "<ts>", "type": "codon.completed", "data": {"codonId": "summarize-notes", "success": true, "cost":"<n>", "duration":"<n>", "exitStatus": {"type": "success"}}}

To watch terminal codon outcomes as they arrive, we'll stream JSONL records and select the event type:

tail -f .hankweave/events/events.jsonl | jq -c 'select(.type == "codon.completed")'

After the run, we can inspect the reported cost at .data.cost:

jq -c 'select(.type == "codon.completed") | {id, codonId: .data.codonId, cost: .data.cost, success: .data.success}' \
  .hankweave/events/events.jsonl

Two summation traps follow from how these events are emitted. Do not sum every token.usage record: that event is emitted for incremental per-call usage and again for the codon's final total. Do not sum every codon.completed record when retries are present: there is one terminal event per attempt, and the final event's cost already includes accumulated retry cost. For a per-run reported-codon-cost view, establish the run boundaries from state.transition, deduplicate retransmitted event IDs, retain the final codon.completed for each (runId, runtime codonId), and sum only that view. Keep continuation runs and loop runtime IDs distinct. This is a recipe for that defined scope, not a captured end-to-end aggregation test; interrupted or skipped work, extensions, sentinels, health checks, and unpriced routes need their own scope decisions.

A complete live execution with one tracked codon recorded its cost separately from provider health checks and sentinel calls:

- Tracked codon cost: $0.10441000 (provider health checks and sentinel calls are separate)

Check it: after a run, confirm that .hankweave/events/events.jsonl exists and that a clean run has a codon.completed record for its terminal codon; a retry adds a record for each failed attempt. The normalized fixture shows the .data.cost path, and the complete execution capture records the tracked codon cost without claiming that it is the provider's invoice.

Upload a finished run to Braintrust or Langfuse#

The journal answers "what happened" as flat records. For a hierarchical view – codons containing LLM calls and tool calls, with costs rolled up – use the standalone hankweave-trace package to transform an execution directory containing the run's state, journal, and per-codon logs into a trace tree; the journal is stored at <exec>/.hankweave/events/events.jsonl. Its documented modes are upload for post-run batches, watch for live spans followed by a final upload, and generate for provider payload JSON on stdout with progress on stderr; generate does not require credentials. These package details are documented since 0.7.0, so verify them against the version you install.

For a post-run upload, select a platform and pass the execution directory. Set one of the credential variables listed in Control credentials, content, and re-uploads before running this command; do not put a real key in a page or shell history:

EXEC_DIR="$HOME/.hankweave-executions/your-execution-id"
npx -y hankweave-trace upload "$EXEC_DIR" --braintrust

The package surface documents --braintrust and --langfuse platform selection, --project <name> for the Braintrust project (default Hankweave), --latest-only, --redact, --force, and comma-separated --tags. Confirm the installed package's help before relying on a flag: this surface is not part of Hankweave's locked runtime artifact.

Once uploaded, a trace has one root for a Hankweave run. Its child spans represent codons, loops, LLM calls, tool calls, rig setup, and sentinels; the root carries run-level cost, duration, status, and tags, while loops group iterations. The exact placement of tokens and costs differs by codon harness; see Trust the cost numbers in a trace before comparing spans.

If credentials are missing, the package tells you so directly. The pinned config capture below shows how that is reported at 0.10.0:

hankweave-trace configuration
==============================

Braintrust: not configured
  Issue:    missing HANKWEAVE_TRACE_BRAINTRUST_API_KEY

Langfuse:   not configured
  Issue:    missing HANKWEAVE_TRACE_LANGFUSE_PUBLIC_KEY and HANKWEAVE_TRACE_LANGFUSE_SECRET_KEY

Config file: none found

Issues:
  - No tracing platform configured. Set HANKWEAVE_TRACE_BRAINTRUST_API_KEY or HANKWEAVE_TRACE_LANGFUSE_* env vars.
npm notice
npm notice New major version of npm available! 10.9.2 -> 12.0.2
npm notice Changelog: https://github.com/npm/cli/releases/tag/v12.0.2
npm notice To update run: npm install -g npm@12.0.2
npm notice
exit=0

Check it: for the standalone package, verify the installed version's upload, watch, and generate behavior rather than treating this page's historical package surface as a locked CLI contract. The captured config output confirms the package's configuration surface, not a successful upload.

Let the runtime upload the trace for you#

Running hankweave-trace by hand works well for a finished run, but it is easy to forget. The runtime can invoke the package itself: at startup it checks the trace configuration, and on shutdown it uploads before exiting.

Set HANKWEAVE_TRACE_BRAINTRUST and/or HANKWEAVE_TRACE_LANGFUSE as presence-based enable flags. They are distinct from the credential variables consumed by hankweave-trace. Under Bun the runtime defaults to bunx hankweave-trace; otherwise it uses npx hankweave-trace. Set HANKWEAVE_TRACE_BINARY when you need a pinned global install or an air-gapped executable.

At startup, the runtime runs <binary> config with a 30-second timeout and prints the configuration block. If the output contains configuration issues, it logs a warning before the run has spent time on codons. On graceful shutdown it synchronously runs the upload before exiting; force-shutdown gets a best-effort call, and an upload guard prevents a second upload. The runtime's upload command always includes --force, so its own re-uploads bypass the manual dedup marker; deterministic (derived consistently from run data) span IDs make that safe.

A real 0.10.0 run with HANKWEAVE_TRACE_BRAINTRUST=1 and no credential shows the startup configuration and the attempted shutdown upload. The attempt fails because the credential is absent; the capture does not demonstrate a successful platform upload:

--- hankweave-trace config ---
hankweave-trace configuration
==============================

Braintrust: not configured
  Issue:    missing HANKWEAVE_TRACE_BRAINTRUST_API_KEY

Langfuse:   not configured
  Issue:    missing HANKWEAVE_TRACE_LANGFUSE_PUBLIC_KEY and HANKWEAVE_TRACE_LANGFUSE_SECRET_KEY

Config file: none found

Issues:
  - No tracing platform configured. Set HANKWEAVE_TRACE_BRAINTRUST_API_KEY or HANKWEAVE_TRACE_LANGFUSE_* env vars.
[hankweave-trace config stderr] npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7
[<ts>] [ERROR] [hankweave-trace config stderr] npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7
[hankweave-trace] WARNING: tracing is misconfigured — upload on exit will likely fail.
[<ts>] [ERROR] [hankweave-trace] WARNING: tracing is misconfigured — upload on exit will likely fail.
Running in headless mode on port <port>
➜ Listening on: http://localhost:<port>/ (all interfaces)
> Uploading trace: npx hankweave-trace upload "~/.hankweave-executions/<exec-id>" --force --braintrust
npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7
Error: --braintrust requires a Braintrust API key (env, config file, or HANKWEAVE_TRACE_BRAINTRUST_API_KEY)
[<ts>] [ERROR] npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7
Error: --braintrust requires a Braintrust API key (env, config file, or HANKWEAVE_TRACE_BRAINTRUST_API_KEY)

Check it: with a tracing flag enabled, look for the --- hankweave-trace config --- block at startup and an upload command containing --force --braintrust at shutdown. The capture shows both, including the absent-credential failure.

If an enabled platform lacks credentials, startup emits this warning in the server log:

41:[<ts>] [INFO] --- hankweave-trace config ---
42:[<ts>] [INFO] hankweave-trace configuration
54:[<ts>] [INFO]   - No tracing platform configured. Set HANKWEAVE_TRACE_BRAINTRUST_API_KEY or HANKWEAVE_TRACE_LANGFUSE_* env vars.
55:[<ts>] [ERROR] [hankweave-trace config stderr] npm warn exec The following package was not found and will be installed: hankweave-trace@0.0.7
56:[<ts>] [ERROR] [hankweave-trace] WARNING: tracing is misconfigured — upload on exit will likely fail.

Check it: treat the warning as a configuration failure to fix before relying on the trace, not as evidence that an upload succeeded. Runtime trace variables are documented with the rest of the environment variables at /reference/environment-variables.

Control credentials, content, and re-uploads#

Whichever upload path you use, the same package-level settings decide where the trace goes and what it contains. These hankweave-trace controls are documented since 0.7.0; verify them against the version you install before relying on a default or config path.

Choose the credentials and content controls for your destination:

• HANKWEAVE_TRACE_BRAINTRUST_API_KEY and HANKWEAVE_TRACE_BRAINTRUST_PROJECT (default Hankweave).
• HANKWEAVE_TRACE_LANGFUSE_PUBLIC_KEY, HANKWEAVE_TRACE_LANGFUSE_SECRET_KEY, and HANKWEAVE_TRACE_LANGFUSE_BASE_URL (default https://cloud.langfuse.com).
• HANKWEAVE_TRACE_TAGS and HANKWEAVE_TRACE_REDACT.

It reads .hankweave-trace.json in the working directory, then ~/.config/hankweave-trace/config.json. Values beginning with $ resolve from the environment. Precedence is CLI flags, environment variables, config file, then defaults. Configure both platforms if you want both to receive the trace; a failure on one does not block the other.

Use --redact or HANKWEAVE_TRACE_REDACT=1 to replace prompts, model output, tool input and output, thinking blocks, and sentinel observations with [redacted]. Span structure, names, metrics, errors, status, tags, and model names remain. Span IDs are deterministic from run data and platforms upsert by ID. Manual re-upload is blocked by .hankweave/tracing-marker.json unless you pass --force.

Trust the cost numbers in a trace#

Trace dashboards show token and cost figures, but where those figures sit – and how the platform computes them – depends on the codon harness and the destination. These hankweave-trace span-placement and cost-display behaviors are documented since 0.7.0; verify them against the version you install.

Claude SDK codons put token counts on individual LLM-call spans, leaving parent spans without those tokens; both platforms sum from children, so summing parent and child tokens double-counts. Pi-harness codons do not expose a per-message breakdown, so their token counts sit on the codon span and child LLM spans show zero.

Braintrust computes its own estimate from tokens, so the trace stores Hankweave's computed cost in metadata.hankweaveCost. Langfuse can overestimate Anthropic costs when prompt caching is involved; the trace overrides each generation's totalCost with the Hankweave-computed value. These are trace values, not a provider invoice. A trace built from the journal inherits the journal's cost fields, including the 0 cost reported by pi/zai/* codons at 0.10.0.

Choose between Braintrust and Langfuse#

The error-visibility and platform-comparison behavior below is documented since 0.7.0; verify it against the version you install.

For this 0.10.0 documentation snapshot, Braintrust is cloud-only and provides generic spans plus AI features such as chart builder and topic maps. Langfuse is open source and can be self-hosted or used as a cloud service; it provides a native generation type and sessions for grouping runs of the same hank (a named execution definition).

Both platforms surface failed runs, failed codons and their failure reasons, tool errors, rig-setup failures, and sentinel error unloads on the relevant spans. Langfuse additionally marks the trace with level: "ERROR" when the run fails. Choose the platform based on hosting and grouping needs, then keep the span-cost caveats above in view.

Tell traces apart from Hankweave telemetry#

Hankweave's product telemetry is a separate pipeline from hankweave-trace. It is enabled by default and sends to the self-hosted PostHog endpoint https://hw-telemetry.southbridge.ai; opt-out precedence is DO_NOT_TRACK=1, then HANKWEAVE_TELEMETRY=0|false, CI auto-detection, and finally hankweave.json with telemetry.enabled=false. The full contract is in /reference/telemetry.

Your trace upload instead runs a separate binary against your execution directory and sends to the Braintrust or Langfuse endpoint selected by your credentials. This is a separate path from product telemetry.