Rolling back when a codon breaks

docs.lock records the exact package, repository commit, schema hashes (the validated data shapes), and surfaces resolved for this documentation set.

An agent run can go wrong in the middle of a task: a codon rewrites the wrong file, a loop drifts, or a shutdown interrupts work that was half-finished. When that happens, the question is how to get the files back to a known state without touching your own project's Git history. Hankweave's answer is a checkpoint system: a shadow Git repository that snapshots selected files at each lifecycle event, and a rollback operation that restores any of those snapshots while keeping every previous timeline inspectable. This page builds that system up piece by piece: where the snapshots live, what they capture, when they are created, how rollback works, and what the system cannot protect.

Why Git sits underneath the agent workspace#

Recovering agent-modified files requires naming, comparing, and restoring snapshots of file state. Git already does all three, so Hankweave keeps a separate Git repository for that purpose rather than inventing its own format. The word separate matters: the checkpoint repository never writes to the operator's .git/, so rolling back a run cannot rewrite the project's history. What this gives us is file-state recovery, not a record of every decision or intermediate thought the agent produced.

A codon is one agent task in a run. Its checkpoint store is a non-bare shadow Git repository at .hankweave/checkpoints/.hankweavecheckpoints inside the execution directory. The store uses agentRoot/–where agent-modified files live–as its work tree. It does not use the operator's .git/ as that work tree.

The unusual repository name is deliberate. .hankweavecheckpoints, rather than .git, prevents Git from detecting the execution directory as a submodule if an operator commits that directory to another repository. The checkpoint repository also has a dedicated .gitconfig in .hankweave/checkpoints/, with the runtime identity Hankweave Runtime <froggie@southbridge.ai> and signing disabled. These settings keep checkpoint Git operations separate from the operator's global Git configuration.

The shipped init fixture shows the outer boundary, the .gitignore an operator's project starts with:

.hankweave/
*.log
node_modules/

That .gitignore excludes .hankweave/, logs, and node_modules/ from the operator's Git view, while those paths remain in the execution directory. The checkpoint work tree is narrower still: it is agentRoot/, not the full execution directory. The diagram below shows how the pieces nest: the operator's repository sits outside, the execution directory holds both the agent's work tree and the checkpoint store.

The shadow repository selects agent files, then preserves file-state history apart from the operator's repository.

↗ Enlarge

operator project .git/
        │
        └── execution directory
              ├── agentRoot/  ◄── shadow Git work tree
              └── .hankweave/checkpoints/
                    ├── .gitconfig
                    └── .hankweavecheckpoints  ◄── non-bare shadow Git store

With the layout in place, the lifecycle of a snapshot is easier to hold as a small model: patterns select files, Git seals them into a commit, and rollback checks a commit out rather than resetting anything.

// Pseudocode, not the implementation.
checkpointedFiles patterns
  → UnifiedFileResolver (expands patterns and respects .gitignore)
  → git add -f
  → git commit --allow-empty
  → sealed checkpoint
  → rollback: git checkout --force <SHA> to detached HEAD
  → resume: new branch from detached HEAD

Because rollback checks out rather than resets, we can return to an earlier checkpoint and try again without losing the older timeline. It remains available when we want to compare the attempts. The rest of this page fills in each stage of that model, starting with what a checkpoint actually captures.

What a checkpoint actually captures#

A checkpoint seals after a codon completes, and a run rolls back to one when later file state needs to be replaced. Between those two moments, the important question is scope: which files the snapshot contains.

A checkpoint does not sweep up every file that changed. It captures files selected by the checkpointedFiles patterns accumulated from codons run so far. The patterns only grow: the checkpoint after codon 3 includes codon 1's patterns, codon 2's patterns, and codon 3's patterns.

The shipped hank–a JSON program file–shows the field in its codon entries, one checkpointedFiles list per codon:

      "checkpointedFiles": ["analysis-haiku.md"],
…
      "checkpointedFiles": ["analysis-gemini.md"],
…
      "checkpointedFiles": ["analysis-pi.md"],
…
      "checkpointedFiles": ["analysis-gpt.md"],

UnifiedFileResolver expands those patterns, applies every .gitignore rule found in the work tree, and always excludes .git/ and the read_only_data_source/ mount inside agentRoot/. A runtime-created ignore rule also excludes rigArchive/; archived output is tracked through a separate manifest rather than Git.

When a top-level codon uses continuationMode: "continue-previous" and the codon it continues from–or the last codon in the preceding loop–has no checkpointedFiles patterns, loading warns with <codon>: <context> doesn't checkpoint any files. Treat that warning as a scope problem. Files changed outside the selected patterns are invisible to checkpoints and cannot be recovered by rolling back.

When a checkpoint seals#

Scope decides what a checkpoint contains; the lifecycle decides when one is created. Hankweave creates a checkpoint for these statuses:

The exit status is conditional: it is created when the shutdown reason is not all codons completed, checkpointing is enabled, and a codon is current. It is not created for the normal all-codons-completed reason. The Git history can therefore contain an exit: commit, while the interrupted-run capture leaves the codon running with exitCode: null and no persisted checkpoint SHA: the transition does not persist an exitCheckpoint, and skipCheckpoint is written only when the codon's state is skipped. An exit: Git status, a checkpoint/event field such as skipped, and a process exit code are different signals; do not use one as the other.

Each checkpoint's Git message has a structured shape, so the log stays readable without extra tooling. Its first line identifies the status, codon, run, and codon name; the body labels the codon and status again, adds a timestamp, and may add a duration:

{status}:{codonId} [run:{runId}] {codonName}

Codon: {codonName}
Status: {status}
Timestamp: {timestamp}
Duration: {duration}ms

The duration line may be omitted. Checkpoints are still created when no tracked file changed: Hankweave uses --allow-empty, so each lifecycle event can leave a trace even when the file contents are identical to the previous commit.

How the checkpoint store keeps history#

Knowing when checkpoints seal leads to the next question: where those commits accumulate, and how to look at them.

At 0.10.0, the shadow repository is initialized during server startup when Git is available. start() calls initializeCheckpoints() before state-manager initialization and before codons run. That path constructs CheckpointGit, creates the store, and makes the initial empty Initial checkpoint setup commit. The addCheckpointPatterns() initialization call is a defensive fallback, not the normal lifecycle.

Each run receives its own shadow-repository branch named after its runId. Hankweave creates that branch on the run's first checkpoint.

Creating a checkpoint follows four operations:

1. Resolve the accumulated patterns to a file list.
2. Reset the Git index–the staging list–in --mixed mode, preserving the working directory.
3. Add the resolved files with git add -f, in batches of 100.
4. Create the Git commit with --allow-empty.

Because the store is a real Git repository, standard Git commands can inspect it. The server.ready event supplies the execution directory in its executionPath field. From that directory, point Git at the non-bare store explicitly:

# Set EXECUTION_PATH to the executionPath from server.ready before running.
cd "$EXECUTION_PATH"
export GIT_DIR=.hankweave/checkpoints/.hankweavecheckpoints
export GIT_WORK_TREE=agentRoot
git log --oneline --all

A pre-0.10.0 execution that still has a legacy .git directory is migrated to .hankweavecheckpoints at the next initialization. If both directories exist, the legacy .git is quarantined under a timestamped name. With --start-new --force, Hankweave renames the old .hankweave/ directory to .hankweave.backup-{timestamp} and migrates legacy .git directories found inside those backups as well.

A completed-run fixture makes the store concrete. The tree below is what .hankweave/checkpoints/ contains after a real run:

.hankweave/checkpoints/
├── .gitconfig
└── .hankweavecheckpoints
    ├── COMMIT_EDITMSG
    ├── config
    ├── description
    ├── HEAD
    ├── hooks
    │   ├── applypatch-msg.sample
    │   ├── commit-msg.sample
    │   ├── fsmonitor-watchman.sample
    │   ├── post-update.sample
    │   ├── pre-applypatch.sample
    │   ├── pre-commit.sample
    │   ├── pre-merge-commit.sample
    │   ├── pre-push.sample
    │   ├── pre-rebase.sample
    │   ├── pre-receive.sample
    │   ├── prepare-commit-msg.sample
    │   ├── push-to-checkout.sample
    │   ├── sendemail-validate.sample
    │   └── update.sample
    ├── index
    ├── info
    │   └── exclude
    ├── logs
    │   ├── HEAD
    │   └── refs
    │       └── heads
    │           ├── main
    │           └── run-<id>
    ├── objects
    │   ├── 1f
    │   │   └── 9624104db901284fdab5dbcb898415c92a88bd
    │   ├── 4b
    │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904
    │   ├── b7
    │   │   └── 6640ebba959f562bf18c19fb9f942ef28f15ca
    │   ├── cb
    │   │   └── b70b93f9e602bd3c7e3af5193dea2ddb827a66
    │   ├── d2
    │   │   └── f04ff1df2bdbbd69db9426a6997923e55ed0ae
    │   ├── info
    │   └── pack
    ├── ORIG_HEAD
    └── refs
        ├── heads
        │   ├── main
        │   └── run-<id>
        └── tags

The .gitconfig sits beside the non-bare .hankweavecheckpoints store. Its HEAD, refs, objects, and a working index are inside that store; the refs include main and a per-run branch. From the execution directory, standard Git can list every branch's history:

git --git-dir=.hankweave/checkpoints/.hankweavecheckpoints log --oneline --all

The completed-run fixture's normalized history shows the two commits a minimal run produces: the initial setup commit, then the codon's completion checkpoint.

<sha7> completed:write-line [run:<id>] Write one line
<sha7> Initial checkpoint setup

How to roll back without losing history#

Everything so far describes how snapshots accumulate. Rollback is the operation that makes them useful, and in hankweave 0.10.0 it is a WebSocket operation, not a CLI flag. The rollback command family has three commands: rollback.toLastSuccess selects the most recent completed checkpoint; rollback.toCodon selects a checkpoint for a codon; and rollback.toCheckpoint selects a particular commit SHA, with partial SHA matching supported. checkpoint.list is a separate command for inspecting the available checkpoints.

For the wider execution-directory layout, see Execution directory. For --start-new and --attach, see CLI reference. For command envelopes, see Protocol.

The command signatures below are the exact wire shapes a client sends:

export interface ListCheckpointsCommand {
  id: string;
  type: "checkpoint.list";
  data?: {
    /** Optional run ID to list checkpoints for */
    runId?: string;
  };
}

/**
 * Rollback to a specific checkpoint.
 */
export interface RollbackToCheckpointCommand {
  id: string;
  type: "rollback.toCheckpoint";
  data: {
    /** Checkpoint SHA (can be partial) */
    checkpointSha: string;
    /** Whether to auto-restart after rollback */
    autoRestart?: boolean;
  };
}

/**
 * Rollback to a codon with specific checkpoint type.
 */
export interface RollbackToCodonCommand {
  id: string;
  type: "rollback.toCodon";
  data: {
    /** Codon ID to rollback to */
    codonId: string;
    /** Checkpoint type within that codon */
    checkpointType: "start" | "end" | "rig-setup" | "completed" | "error" | "skipped";
    /** Whether to auto-restart after rollback */
    autoRestart?: boolean;
  };
}

/**
 * Rollback to last successful codon.
 */
export interface RollbackToLastSuccessCommand {
  id: string;
  type: "rollback.toLastSuccess";
  data?: {
    /** Whether to auto-restart after rollback */
    autoRestart?: boolean;
  };
}

For rollback.toCodon, start and end are selection aliases, not stored checkpoint types. start selects the first available checkpoint, preferring rig-setup (the setup-stage checkpoint), then completed, error, and skipped; end selects the last available, preferring completed, error, skipped, and rig-setup. The command therefore lands on the first type in the relevant order that exists for that codon.

A WebSocket client triggers rollback by sending one of these commands. Rollback cannot begin while a codon is running; the error advises using codon.forceStop first. Once rollback begins, state-changing WebSocket commands are blocked. checkpoint.list, server.shutdown, server.force_shutdown, ping, and history.sync remain available; ping.broadcast is not exempted.

File restoration uses git checkout --force to a detached HEAD state–Git points at the target commit without moving a branch. That restores the target working-directory state without destroying the old branch's history, which is why this path is used instead of git reset --hard. When execution resumes, Hankweave creates a new branch from the detached HEAD. The old timeline remains intact, and a SHA search covers checkpoints from all historical runs in state.json, including previously rolled-back timelines.

Because rigArchive/ is outside Git and its entries are tracked through a separate manifest, files moved there after the target checkpoint are restored to agentRoot/ during rollback. The workspace therefore includes the checkpoint's tracked-file state and the archive restoration needed to match that point in time.

What a checkpoint cannot save you from#

Rollback is powerful within its scope, and that scope has hard edges worth knowing before you rely on it.

The shadow repository isolates run state from the operator's project. It lives under the execution directory's .hankweave/, uses its own Git configuration, and does not write to the operator's .git/ or global configuration. The shipped fixture excludes .hankweave/ from the operator's Git view.

That isolation is not continuous backup. A checkpoint is a snapshot of the tracked files at one point in time. If a tracked file is deleted between checkpoints, the resolver no longer returns it for the add step, so the deletion is not recorded: the later checkpoint still contains its last committed content, and rolling back to that checkpoint restores the copy. A file that never matched a pattern cannot be recovered by any rollback.

A checkpoint also does not make model output repeatable. Model output is stochastic. The guarantee is inspectable and restorable file state, not repeatable generation.

Only files selected by checkpointedFiles and left in scope after .gitignore filtering are tracked. Choose those patterns for the work you will need to inspect or recover; intermediate states and decisions are not automatically preserved.

How to observe checkpoint protocol events#

Within those limits, the protocol gives clients a live view of what the checkpoint system is doing. The checkpoint.list WebSocket command emits a checkpoint.list event containing runId, checkpoints, and currentBranch. Each checkpoint entry carries codonId, codonName, checkpointType (one of rig-setup, completed, error, or skipped–never exit), sha, status, and timestamp. Checkpoint and rollback events are journaled and can be routed to sentinels; the complete payload catalog belongs to the events reference.

The table below lists the checkpoint and rollback events with their payload fields and schema receipts, so you can trace each event from emission to validation:

In --headless mode (without the terminal interface), the interaction surface changes for CI/CD and scripts, but checkpoint and rollback events continue to be journaled and emitted. The command and event details remain in the generated protocol and event references.

How to inspect and debug a checkpoint#

Events tell you what happened as it happens. When a run already looks wrong and you need to reconstruct it after the fact, use this order rather than guessing from filenames: read .hankweave/state.json; follow the codon's exact claudeLogPath value from that state; correlate journal events by codonId; then diff the relevant checkpoint SHAs to identify the file changes.

state.json lives at .hankweave/state.json inside the execution directory. Each codon record can persist a checkpoint SHA under rigSetupCheckpoint, completionCheckpoint, errorCheckpoint, or skipCheckpoint. The type also declares an optional initialCheckpointSha inside the fresh startingConditions variant–not as a top-level run field–but normal 0.10.0 startup does not write it: a fresh captured run contains only startingConditions: { type: "fresh" }. The initial empty-commit SHA is available through explicit Git inspection or a rollback.toCheckpoint partial-SHA search, not through an automatically populated field.

When inspecting the Git log, an exit: prefix identifies the Git status of a shutdown checkpoint. It is not a checkpointType, and neither exit: nor skipped is a process exit code. For the full procedure, including claudeLogPath details, continue to Observe and debug. The execution-directory layout belongs to Execution directory; journal events are persisted by EventJournal.

If you attach the terminal interface to an already-running server with --attach, it is read-only. It can observe checkpoint events, but it cannot issue rollback commands.