Rolling back when a codon breaks

docs.lock records the exact package, repository commit, schema hashes (the validated data shapes), and surfaces resolved for this documentation set.

An agent run can go wrong in the middle of a task: a codon rewrites the wrong file, a loop drifts, or a shutdown interrupts work that was half-finished. When that happens, the question is how to get the files back to a known state without touching your own project's Git history. Hankweave's answer is a checkpoint system: a shadow Git repository that snapshots selected files at each lifecycle event, and a rollback operation that restores any of those snapshots while keeping every previous timeline inspectable. This page builds that system up piece by piece: where the snapshots live, what they capture, when they are created, how rollback works, and what the system cannot protect.

Why Git sits underneath the agent workspace#

Recovering agent-modified files requires naming, comparing, and restoring snapshots of file state. Git already does all three, so Hankweave keeps a separate Git repository for that purpose rather than inventing its own format. The word separate matters: the checkpoint repository never writes to the operator's .git/, so rolling back a run cannot rewrite the project's history. What this gives us is file-state recovery, not a record of every decision or intermediate thought the agent produced.

A codon is one agent task in a run. Its checkpoint store is a non-bare shadow Git repository at .hankweave/checkpoints/.hankweavecheckpoints inside the execution directory. The store uses agentRoot/–where agent-modified files live–as its work tree. It does not use the operator's .git/ as that work tree.

Git is not an optional dependency here. Checkpoints, rollback, and crash recovery are all built on the shadow repository, so the CLI proves git --version runs before it creates, wipes, or copies into the execution directory, and the runtime repeats the check when it initializes checkpoints. There is no opt-out: a missing Git exits with GitUnavailableError and the execution directory is left untouched (though the CLI's $schema auto-add can already have rewritten a schema-less hank.json on disk).

The unusual repository name is deliberate. .hankweavecheckpoints, rather than .git, prevents Git from detecting the execution directory as a submodule if an operator commits that directory to another repository. The checkpoint repository also has a dedicated .gitconfig in .hankweave/checkpoints/, with the runtime identity Hankweave Runtime <froggie@southbridge.ai> and signing disabled. These settings keep checkpoint Git operations separate from the operator's global Git configuration.

The shipped init fixture shows the outer boundary, the .gitignore an operator's project starts with:

.hankweave/
*.log
node_modules/

That .gitignore excludes .hankweave/, logs, and node_modules/ from the operator's Git view, while those paths remain in the execution directory. The checkpoint work tree is narrower still: it is agentRoot/, not the full execution directory. The diagram below shows how the pieces nest: the operator's repository sits outside, the execution directory holds both the agent's work tree and the checkpoint store.

The shadow repository selects agent files, then preserves file-state history apart from the operator's repository.

↗ Enlarge

operator project .git/
        │
        └── execution directory
              ├── agentRoot/  ◄── shadow Git work tree
              └── .hankweave/checkpoints/
                    ├── .gitconfig
                    └── .hankweavecheckpoints  ◄── non-bare shadow Git store

With the layout in place, the lifecycle of a snapshot is easier to hold as a small model: patterns select files, Git seals them into a commit, and rollback checks a commit out rather than resetting anything.

↗ Enlarge

// Pseudocode, not the implementation.
checkpointedFiles patterns
  → UnifiedFileResolver (expands patterns and respects .gitignore)
  → git add -f
  → git commit --allow-empty
  → sealed checkpoint
  → rollback: git checkout --force <SHA> to detached HEAD
  → resume: new branch from detached HEAD

Because rollback checks out rather than resets, we can return to an earlier checkpoint and try again without losing the older timeline. It remains available when we want to compare the attempts. The rest of this page fills in each stage of that model, starting with what a checkpoint actually captures.

What a checkpoint actually captures#

A checkpoint seals after a codon completes, and a run rolls back to one when later file state needs to be replaced. Between those two moments, the important question is scope: which files the snapshot contains.

A checkpoint does not sweep up every file that changed. It captures files selected by the checkpointedFiles patterns accumulated from codons run so far. The patterns only grow: the checkpoint after codon 3 includes codon 1's patterns, codon 2's patterns, and codon 3's patterns. The patterns are read from the execution plan rather than the top-level codon list, so a loop iteration (runtime id <codonId>#<iteration>) registers and restores its own checkpointedFiles across a restart.

The shipped hank–a JSON program file–shows the field in its codon entries, one checkpointedFiles list per codon:

      "checkpointedFiles": ["analysis-haiku.md"],
…
      "checkpointedFiles": ["analysis-gemini.md"],
…
      "checkpointedFiles": ["analysis-pi.md"],
…
      "checkpointedFiles": ["analysis-gpt.md"],

UnifiedFileResolver expands those patterns, applies every .gitignore rule found in the work tree, and always excludes .git/ and the read_only_data_source/ directory the runtime links into agentRoot/. rigArchive/ never enters a checkpoint either, and the reason is location, not an ignore rule: the checkpoint work tree is agentRoot/, and rigArchive/ is that directory's sibling at the execution root, outside the work tree. (The runtime does write a rigArchive/ line into a .gitignore at the execution root, but that file is not what excludes it.) Archived output is tracked through a separate manifest rather than Git.

The same resolver also governs tool-path watched-file matching: a file.updated event is emitted only when the written path matches the codon's checkpointedFiles under those same glob and ignore semantics, where a slash-less pattern matches only at the work-tree root.

When a top-level codon uses continuationMode: "continue-previous" and the codon it continues from–or the last codon in the preceding loop–has no checkpointedFiles patterns, loading warns with <codon>: <context> doesn't checkpoint any files. Treat that warning as a scope problem. Files changed outside the selected patterns are invisible to checkpoints and cannot be recovered by rolling back.

When a checkpoint seals#

Scope decides what a checkpoint contains; the lifecycle decides when one is created. A checkpoint records one of four stored types:

The table's exit row needs care, because exit is a Git commit status, not a stored checkpoint type. Only a graceful shutdown writes it, when a codon is still current and the reason is not all codons completed–a first SIGINT or SIGTERM, a client server.shutdown, or a codon-failure shutdown. A force shutdown, a second signal, a crash, or a kill writes no checkpoint at all. When the exit status is written, the runtime maps it to the stored type skipped (every status other than rig-setup, completed, or error maps there), and skipCheckpoint is persisted only when the codon's state is skipped. The interrupted-run capture therefore leaves the codon running with exitCode: null and no persisted checkpoint SHA. An exit: Git status, a stored checkpoint type, and the process exit code are three different signals; do not use one as another. The process exit code follows the shutdown reason: 0 for all codons completed when the run did not fail, and for the exact reasons SIGINT, SIGTERM, and client request; every other reason exits 1, and a force shutdown always exits 1.

Each checkpoint's Git message has a structured shape, so the log stays readable without extra tooling. Its first line identifies the status, codon, run, and codon name; the body labels the codon and status again, adds a timestamp, and may add a duration:

{status}:{codonId} [run:{runId}] {codonName}

Codon: {codonName}
Status: {status}
Timestamp: {timestamp}
Duration: {duration}ms

The duration line may be omitted. Checkpoints are still created when no tracked file changed: Hankweave uses --allow-empty, so each lifecycle event can leave a trace even when the file contents are identical to the previous commit.

How the checkpoint store keeps history#

Knowing when checkpoints seal leads to the next question: where those commits accumulate, and how to look at them.

The shadow repository is initialized during server startup, not lazily. start() calls initializeCheckpoints() before state-manager initialization and before codons run. That path asserts Git is available, constructs CheckpointGit, builds the store, and makes the initial empty Initial checkpoint setup commit. addCheckpointPatterns() has no initialize fallback; it only registers patterns.

Each run receives its own shadow-repository branch named after its runId. Hankweave creates that branch on the run's first checkpoint.

Creating a checkpoint follows four operations:

1. Resolve the accumulated patterns to a file list.
2. Reset the Git index–the staging list–in --mixed mode, preserving the working directory.
3. Add the resolved files with git add -f, in batches of 100.
4. Create the Git commit with --allow-empty.

The store is built to survive a crash. A fresh repository is constructed in a temp folder and renamed into place only after its first commit exists; at boot, leftover temp folders and stale Git locks are removed, a store whose HEAD does not resolve is rebuilt, and a repository Git cannot read stops the boot rather than reading as empty. When recovery finds nothing restorable, the runtime snapshots the work tree to a recovery/<timestamp> branch, emits a "Recovery degraded" event, and starts a new run.

Because the store is a real Git repository, standard Git commands can inspect it. The server.ready event supplies the execution directory in its executionPath field–never assume a fixed prefix. From that directory, point Git at the non-bare store explicitly:

# Set EXECUTION_PATH to the executionPath from server.ready before running.
cd "$EXECUTION_PATH"
export GIT_DIR=.hankweave/checkpoints/.hankweavecheckpoints
export GIT_WORK_TREE=agentRoot
git log --oneline --all

An execution that still has a legacy .git directory is migrated to .hankweavecheckpoints at the next initialization. If both directories exist, the legacy .git is quarantined under a timestamped name. With --start-new --force, Hankweave renames the old .hankweave/ directory to .hankweave.backup-{timestamp} and migrates legacy .git directories found inside those backups as well.

A completed-run fixture makes the store concrete. The tree below is what .hankweave/checkpoints/ contains after a real run:

.hankweave/checkpoints/
├── .gitconfig
└── .hankweavecheckpoints
    ├── COMMIT_EDITMSG
    ├── HEAD
    ├── ORIG_HEAD
    ├── config
    ├── description
    ├── hooks
    │   ├── applypatch-msg.sample
    │   ├── commit-msg.sample
    │   ├── fsmonitor-watchman.sample
    │   ├── post-update.sample
    │   ├── pre-applypatch.sample
    │   ├── pre-commit.sample
    │   ├── pre-merge-commit.sample
    │   ├── pre-push.sample
    │   ├── pre-rebase.sample
    │   ├── pre-receive.sample
    │   ├── prepare-commit-msg.sample
    │   ├── push-to-checkout.sample
    │   ├── sendemail-validate.sample
    │   └── update.sample
    ├── index
    ├── info
    │   └── exclude
    ├── logs
    │   ├── HEAD
    │   └── refs
    │       └── heads
    │           ├── main
    │           └── run-<id>
    ├── objects
    │   ├── 25
    │   │   └── f29bc93e28a428a30db4e8a21491bc86464654
    │   ├── 4b
    │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904
    │   ├── b7
    │   │   └── 6640ebba959f562bf18c19fb9f942ef28f15ca
    │   ├── cb
    │   │   └── b70b93f9e602bd3c7e3af5193dea2ddb827a66
    │   ├── eb
    │   │   └── 22234b6505ca796ed73627719b8cad4afd4b67
    │   ├── info
    │   └── pack
    └── refs
        ├── heads
        │   ├── main
        │   └── run-<id>
        └── tags

The .gitconfig sits beside the non-bare .hankweavecheckpoints store. Its HEAD, refs, objects, and a working index are inside that store; the refs include main and a per-run branch. From the execution directory, standard Git can list every branch's history:

git --git-dir=.hankweave/checkpoints/.hankweavecheckpoints log --oneline --all

The completed-run fixture's normalized history shows the two commits a minimal run produces: the initial setup commit, then the codon's completion checkpoint.

<sha7> completed:write-line [run:<id>] Write one line
<sha7> Initial checkpoint setup

How to roll back without losing history#

Everything so far describes how snapshots accumulate. Rollback is the operation that makes them useful, and it is a WebSocket operation, not a CLI flag–no CLI flag exists for it. The rollback command family has three commands. rollback.toLastSuccess targets the completion checkpoint of the newest completed codon, when Git confirms it; failing that, the newest codon with any Git-confirmed checkpoint; and with nothing restorable, it snapshots the work tree to a recovery branch and continues degraded. rollback.toCodon selects a checkpoint for a codon. rollback.toCheckpoint selects a particular commit SHA, with partial SHA matching supported, and searches across all historical runs in state.json, so checkpoints from previously rolled-back timelines are reachable. checkpoint.list is a separate command for inspecting the available checkpoints.

For the wider execution-directory layout, see Execution directory. For --start-new and --attach, see CLI reference. For command envelopes, see Protocol.

The command signatures below are the exact wire shapes a client sends:

export interface ListCheckpointsCommand {
  id: string;
  type: "checkpoint.list";
  data?: {
    /** Optional run ID to list checkpoints for */
    runId?: string;
  };
}

/**
 * Rollback to a specific checkpoint.
 */
export interface RollbackToCheckpointCommand {
  id: string;
  type: "rollback.toCheckpoint";
  data: {
    /** Checkpoint SHA (can be partial) */
    checkpointSha: string;
    /** Whether to auto-restart after rollback */
    autoRestart?: boolean;
  };
}

/**
 * Rollback to a codon with specific checkpoint type.
 */
export interface RollbackToCodonCommand {
  id: string;
  type: "rollback.toCodon";
  data: {
    /** Codon ID to rollback to */
    codonId: string;
    /** Checkpoint type within that codon */
    checkpointType: "start" | "end" | "rig-setup" | "completed" | "error" | "skipped";
    /** Whether to auto-restart after rollback */
    autoRestart?: boolean;
  };
}

/**
 * Rollback to last successful codon.
 */
export interface RollbackToLastSuccessCommand {
  id: string;
  type: "rollback.toLastSuccess";
  data?: {
    /** Whether to auto-restart after rollback */
    autoRestart?: boolean;
  };
}

For rollback.toCodon, start and end are selection aliases, not stored checkpoint types. start selects the first available checkpoint, preferring rig-setup (the setup-stage checkpoint), then completed, error, and skipped; end selects the last available, preferring completed, error, skipped, and rig-setup. The command therefore lands on the first type in the relevant order that exists for that codon.

A WebSocket client triggers rollback by sending one of these commands. Rollback cannot begin while a codon is running; the error advises using codon.forceStop first. Once rollback begins, state-changing WebSocket commands are blocked. checkpoint.list, server.shutdown, server.force_shutdown, ping, and history.sync remain available; ping.broadcast is not exempted.

File restoration uses git checkout --force to a detached HEAD state–Git points at the target commit without moving a branch. That restores the target working-directory state without destroying the old branch's history, which is why this path is used instead of git reset --hard. When execution resumes, Hankweave creates a new branch from the detached HEAD. The old timeline remains intact and accessible.

Because rigArchive/ sits outside the work tree and its entries are tracked through a separate manifest, rollback restores archive entries that were archived strictly after the target on the abandoned line back into agentRoot/. Selection is by Git reachability between the target and the pre-rollback HEAD, not manifest list order; entries at or before the target, or on unrelated timelines, stay archived, and only successfully restored entries leave the manifest. The workspace therefore includes the checkpoint's tracked-file state and the archive restoration needed to match that point in time.

What a checkpoint cannot save you from#

Rollback is powerful within its scope, and that scope has hard edges worth knowing before you rely on it.

The shadow repository isolates run state from the operator's project. It lives under the execution directory's .hankweave/, uses its own Git configuration, and does not write to the operator's .git/ or global configuration. The shipped fixture excludes .hankweave/ from the operator's Git view.

That isolation is not continuous backup. A checkpoint is a snapshot of the tracked files at one point in time. If a tracked file is deleted between checkpoints, the resolver no longer returns it for the add step, so the deletion is not recorded: the later checkpoint still contains its last committed content, and rolling back to that checkpoint restores the copy. A file that never matched a pattern cannot be recovered by any rollback.

A checkpoint also does not make model output repeatable. Model output is stochastic. The guarantee is inspectable and restorable file state, not repeatable generation.

Only files selected by checkpointedFiles and left in scope after .gitignore filtering are tracked. Choose those patterns for the work you will need to inspect or recover; intermediate states and decisions are not automatically preserved.

How to observe checkpoint protocol events#

Within those limits, the protocol gives clients a live view of what the checkpoint system is doing. The checkpoint.list WebSocket command replies with an event containing runId, checkpoints, and currentBranch. A supplied runId filters the list to that run; an omitted one lists every run's checkpoints, most recent first. Each checkpoint entry carries codonId, codonName, checkpointType (one of rig-setup, completed, error, or skipped–never exit), sha, status, and timestamp. Checkpoint and rollback events are journaled and can be routed to sentinels; the complete payload catalog belongs to the events reference.

The table below lists the checkpoint and rollback events with their payload fields and schema receipts, so you can trace each event from emission to validation:

In --headless mode (without the terminal interface), the interaction surface changes for CI/CD and scripts, but checkpoint and rollback events continue to be journaled and emitted. The command and event details remain in the generated protocol and event references.

How to inspect and debug a checkpoint#

Events tell you what happened as it happens. When a run already looks wrong and you need to reconstruct it after the fact, use this order rather than guessing from filenames: read .hankweave/state.json; follow the codon's exact claudeLogPath value from that state; correlate journal events by codonId; then diff the relevant checkpoint SHAs to identify the file changes.

state.json lives at .hankweave/state.json inside the execution directory. Each codon record can persist a checkpoint SHA under rigSetupCheckpoint, completionCheckpoint, errorCheckpoint, or skipCheckpoint. The schema also declares a root initialCheckpoint field and an optional initialCheckpointSha inside the fresh startingConditions variant, but the runtime never writes either: a fresh captured run contains only startingConditions: { type: "fresh" }. The initial empty-commit SHA is available through explicit Git inspection or a rollback.toCheckpoint partial-SHA search, not through an automatically populated field.

When inspecting the Git log, an exit: prefix identifies the Git status of a shutdown checkpoint. It is not a stored checkpoint type, and neither exit: nor skipped is a process exit code. For the full procedure, including claudeLogPath details, continue to Observe and debug. The execution-directory layout belongs to Execution directory; journal events are persisted by EventJournal.

If you attach the terminal interface to an already-running server with --attach, it is read-only. It can observe checkpoint events, but it cannot issue rollback commands.