A hank is a sealed sequence of codon tasks; a codon is one bounded agent task in that sequence. Once a hank runs headless, nobody is watching it decide. Some decisions in a workload are safe for an agent to settle on its own, and some are not: a quote that expired before the award date, a handwritten discount attached to a bid, a supplier whose format nobody has seen before. If the agent quietly resolves those, the output looks complete and the mistake is invisible.
This page is a pattern for keeping those calls visible. The idea is simple: decide up front which choices the hank may make, record everything unresolved in durable files that the next codon or a human reviewer can inspect, and reserve a structured escalation format for the calls code cannot defensibly make. The running example is the quote-template-unification anchor hank, which reconciles quotes from eight suppliers against a five-part bill of materials and carries one planted judgment call (JC-1) through to human sign-off.
Map decisions before you freeze the hank#
The pattern starts before any codon runs. Before freezing a hank's task sequence, map the decisions in four columns:
- May make: decisions the hank may settle.
- Must surface: decisions it must show to a person.
- Who resolves: the person or role that settles each surfaced decision.
- Where durable: the file or other surface where the answer remains available.
That map places human judgment at both ends of the run. At authoring time, shape inputs interactively; then run the hank headless–that is, without interactive guidance. At the end, write a judgment-calls review record containing the options, tentative decision, confidence, and the consequence if the decision is wrong. Have a person review it and feed the outcome into the next run. The general pattern has no required filename, so choose one for your hank and document that choice.
The anchor applies this at its start judgment: hand-transcribe the request-for-quotation (RFQ) contract into reference/rfq-bom-rev-b.json from the corpus XLSX/PDF, then ship that JSON inside the hank. Do not ask a codon to parse the binary contract at runtime; the judgment prompts read the transcribed JSON during the run. Here, the anchor is the quote-template-unification sample hank used by the examples.
Use the anchor as a low-density example: it carries one planted judgment call (JC-1) across eight quoting suppliers. For the judgment-density fit boundary, see the fit guide; see the repair tutorial for the call's build walkthrough.
Stop continue from hiding unresolved work#
With the decision map in place, the first enforcement point is the exception queue. Make it a deliverable, not a log. We ask the agent to record unresolved material before moving past a problem, so a person can still find it. That is the intent of the prompt's Continue rule, not a guarantee that a run cannot be interrupted:
Continue must never mean silent omission: if a line, a part, or a whole quote doesn't cleanly resolve, it must appear in `exception-ledger.json` with a reason. It may never simply vanish from your output.
The final award-writing task also preserves one line per ledger exception. A row may be resolved, routed, or marked unresolved, but it must not be dropped, merged away, or summarized out of existence.
Give every standard ledger row these seven fields:
exception_id, type, supplier_code, buyer_part_id, detail, resolution, source_ref
Number exception_id as EX-01, EX-02, and so on in emission order. Pipe-join identifiers when one exception spans multiple parts or blocks. Keep the ledger append-only across codons: preserve earlier rows and append new ones.
Make coverage gaps visible. A missing supplier-by-part combination emits MISSING_COVERAGE and still receives an UNRESOLVED row in unified-records.csv; a missing supplier is a loud failure, not a silent skip.
Apply the same discipline to each hazard. The supplier names below–Aster, Cedar, Fjord, Granite, and Iris–are records in the quote-template-unification anchor; each illustrates a different kind of exception.
- Quarantine an unrecognized intake as
UNKNOWN_TEMPLATE, with zero extracted rows and an onboarding note. Granite's first-time XLSX is quarantined rather than force-fit into a template. - Fail closed on Fjord's
'TBD'prices: usestatus: UNRESOLVED, null prices, andranking_eligible: false. ANO_BIDline is not a zero-price quote. - Log mechanical conflicts. Aster's later
issued_atsubmission wins deduplication, while the one-cent stated-total drift remains aDEDUP_CONFLICTrow. - Route Iris's conditional handwritten offer as
NOTE_ROUTED, carrying the text verbatim. Leave the quoted line unchanged; the offer is a note for a person, not a price.
The captured excerpt below carries the queue's header contract and the first four rows in emission order; EX-04, Cedar's expired-validity exception, is the judgment row, with all five part identifiers pipe-joined into buyer_part_id and a show-flagged resolution that keeps the quote visible but ineligible.
exception_id,type,supplier_code,buyer_part_id,detail,resolution,source_ref
EX-01,UNKNOWN_TEMPLATE,GRN,,No known template exists for Granite Bid Co's internal layout (not NC-RFQ-0042 compliant). Extraction cannot be performed without establishing a template mapping. Supplier onboarding required to define column mappings and data extraction rules.,quarantined,source-quotes/granite-bid.xlsx
EX-02,DEDUP_CONFLICT,AST,NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A,"Two Aster submissions share dedup_key 'AST|NC-RFQ-0042|B'. Original (doc-aster-qb-1047, issued 2026-01-10T09:00:00) stated_total_minor: 604000. Rev2 (doc-aster-qb-1047-rev2, issued 2026-01-14T11:30:00, supersedes doc-aster-qb-1047) stated_total_minor: 604001. Differs by 1 cent. Rev2 wins by later issued_at.",rev2 selected; original superseded and dropped,doc-aster-qb-1047|doc-aster-qb-1047-rev2
EX-03,NO_BID,CDR,NC-1004-A,Cedar declined to bid on NC-1004-A (gasket sheet). status_hint is NO_BID.,line excluded from ranking; unit price and extension set null,doc-cedar-cw-77-b4
EX-04,EXPIRED_VALIDITY,CDR,NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A,Cedar quote valid_until 2026-01-20 precedes RFQ award_decision_date 2026-02-01. The quote is expired at the moment of award.,show-flagged,doc-cedar-cw-77
Check-it 1: After validate-and-repair, inspect exception-ledger.json. In the captured anchor output it is a JSON array of eight rows, EX-01 through EX-08, covering eight exception types. EX-01 is Granite's unknown-template quarantine, EX-02 is Aster's dedup conflict, EX-03 is Cedar's NO_BID row for NC-1004-A, and EX-04 is Cedar's expired-validity call across all five parts; the judgment fields appear on EX-04 only.
Record the call code cannot automate#
Some exceptions are mechanical: a conflict to log, a gap to name. Others are genuine choices that people may reasonably contest, and those need more than a standard row. The shipped anchor uses a four-field judgment-call block in addition to the standard row, and omits those fields from every other row:
judgment_call: true
options: ["exclude", "show-flagged"]
tentative_resolution: "show-flagged"
confidence: 0.5
For JC-1, compare Cedar's valid_until (2026-01-20) with the RFQ award_decision_date (2026-02-01). Because the quote is expired at award time, set ranking_eligible: false on all five Cedar lines, regardless of each line's own status. Emit one EXPIRED_VALIDITY row for the whole quote, with the five part identifiers pipe-joined.
Keep the quote visible but unable to win ranking under the tentative show-flagged resolution. We leave the question open for a human override instead of hiding it in an automatic choice. Confidence 0.5 is deliberate uncertainty, not a confident automatic call. Review instructions may ask what happens if the choice is wrong; in this fixture's four-field shape, keep that consequence in detail and resolution rather than adding a fifth field.
The captures below show the call at three stages: the ledger row with its four judgment fields, the prompt step that instructs the agent to emit it, and the award-brief line that carries it to a reviewer.
{
"exception_id": "EX-04",
"type": "EXPIRED_VALIDITY",
"supplier_code": "CDR",
"buyer_part_id": "NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A",
"detail": "Cedar quote valid_until 2026-01-20 precedes RFQ award_decision_date 2026-02-01. The quote is expired at the moment of award.",
"resolution": "show-flagged",
"source_ref": "doc-cedar-cw-77",
"judgment_call": true,
"options": ["exclude", "show-flagged"],
"tentative_resolution": "show-flagged",
"confidence": 0.5
},
## Step 3 — Quote-level judgment call (JC-1)
Compare each quote's `quote_meta.valid_until` to the RFQ contract's `award_decision_date` (`2026-02-01`). Cedar's `valid_until` is `2026-01-20` — **before** the award date. The quote is expired at the moment of award.
This is a **planted, irresolvable judgment call**, not a bug to silently fix. Whether an expired-at-award quote should be excluded entirely or kept visible-but-ineligible is a human decision. Resolve it as follows and make the judgment visible, not buried:
- Set `ranking_eligible: false` on **every** line of the expired quote, regardless of that line's own per-line status.
- Emit exactly one exception of `type: "EXPIRED_VALIDITY"` for the whole quote (one row, `buyer_part_id` covering all 5 parts pipe-joined, e.g. `NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A`), with these additional fields beyond the standard ones:
- `judgment_call: true`
- `options: ["exclude", "show-flagged"]`
- `tentative_resolution: "show-flagged"` — the quote stays visible in the unified output and cannot win ranking, but a human reviewer can override this.
- `confidence: 0.5` (deliberately uncertain — this is not a confident automatic call).
- EX-04 (EXPIRED_VALIDITY, CDR): Cedar's quote validity period ended before the RFQ award decision date. This judgment call requiring human sign-off places the entire submission in show-flagged status, rendering Cedar's quotations visible but ineligible for winning any part. (source_ref: `doc-cedar-cw-77`).
Treat that captured assessment as the sign-off banner, not as the row-level eligibility source. The structured records distinguish Cedar's four QUOTED rows excluded by EX-04 from its NO_BID row for NC-1004-A, recorded as EX-03.
The held-out oracle encodes the same call under its own local row identifier, EX-01, and names it JC-1:
EX-01,EXPIRED_VALIDITY,CDR,NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A,CDR valid_until 2026-01-20 < award decision date 2026-02-01,"JC-1: planted irresolvable judgment call — exclude-vs-show-flagged is a human decision; quote stays visible, cannot win ranking",doc-cedar-cw-77
Row identifiers are local to each file; the call name is the stable label for the decision. Keep this answer key out of prompts and agent input directories.
Check-it 2: Confirm that the lossless exceptions.csv has the same header as the held-out answer key, that its expired-validity row carries the EXPIRED_VALIDITY identity and source reference, that the ledger row (EX-04) carries the four judgment fields, and that the award report calls for human sign-off.
Keep rendering from making new judgments#
Judgment belongs in the earlier stages and in human review. Make the final rendering task cite and format settled records. It must make no new judgment calls: every fact it states must already be settled in unified-records.csv or exception-ledger.json.
Require a [doc:block] citation for every dollar figure. Name Cedar's four ineligible QUOTED bids and cite EX-04; for NC-1004-A, show the NO_BID exception EX-03 rather than describing that row as an expired bid. Do not omit the quoted bids because they cannot rank. If no row for a part is ranking_eligible, say that no eligible winner exists.
Keep the capability boundary in the hank. validate-and-repair and reconcile use pi/baseten/deepseek-ai/DeepSeek-V4-Pro for judgment work. At the final stage, setup rigs–deterministic setup scripts the hank runs around its codons–render the financial section and serialize exceptions.csv; the haiku agent in award-brief writes only the qualitative assessment from settled records. We reserve the decision-making for the earlier stages and human review.
Check-it 3: Read the rendered outputs together. The captured unified-records.csv has 40 supplier-part data rows–eight quoting suppliers across five BOM parts, or 41 lines including the header–and the ledger has one row for each of the eight exception types. The held-out oracle comparison matches those records and exception types. The corpus plants ten dialect hazards (D1–D10); seven of them produce the eight rows, and D1's quantity breaks, D2's pack pricing, and D4's concatenated part id produce none.
Feed review into the next run#
Use a judgment-calls file as the post-run review surface for a headless hank. In this anchor, the EX-04 row in exception-ledger.json is that review record; its four-field shape keeps the judgment and its consequence durable without making that filename a general default. Have a person review each option, tentative decision, and confidence, then carry the outcome into the next run. The anchor evidence does not prescribe a filename or location for other hanks, so document your convention rather than relying on an implied default.
Keep the held-out answer key out of prompts. The run's data directory contains task inputs only; compare unified-records.csv and exceptions.csv with the answer key after the run, outside the agent's input scope.
Add a sentinel–an event observer that reports without editing–to the validation boundary. The anchor's quality-observer fires after validate-and-repair completes. It reports only status, duration, failure reason, and budget fields present in the completion event. It cannot inspect artifacts, establish row coverage, or infer missing files from absent file-update events. For sentinel configuration details, see the sentinel configuration reference.
The sentinel configuration below wires that boundary: the event expression passes the trigger events into the prompt, and the prompt text itself instructs the model to state what it cannot verify.
{
"id": "quality-observer",
"name": "Quality Observer",
"description": "Observes the validate-and-repair completion event. Reports only status and budget fields present in that event; cannot read or verify output artifacts.",
"trigger": {
"type": "event",
"on": [
"codon.completed"
],
"conditions": [
{
"operator": "equals",
"path": "codonId",
"value": "validate-and-repair"
}
]
},
"execution": {
"strategy": "immediate"
},
"model": "anthropic/claude-haiku-4-5",
"userPromptText": "The codon `validate-and-repair` completed. Here are the triggering completion events:\n\n<%= JSON.stringify(it.events, null, 1) %>\n\nSummarize only status, failure reasons and budget fields actually present. These completion events do not provide file contents or file.updated events. State explicitly that artifact existence, row coverage and correctness cannot be verified from this input. Never treat missing file-update events as proof of missing files. Observe and report only.",
"output": {
"format": "text",
"file": "quality-observer.log"
}
}
The captured report is the current 2026-09-27 capture. It summarizes its completion-event input, not artifact correctness: it confirms the completion status and duration, then states its own limits.
---
# Codon Completion Summary
**Status:** Success
**Cost:** $<cost>
**Duration:** 117,863 ms (~1m 58s)
---
## Verification Limitations
Artifact existence, row coverage, and correctness **cannot be verified** from this input. No file-update events were provided. The absence of file-update events does not indicate missing files.
Check-it 4: After validate-and-repair, open the sentinel output at <execution>/.hankweave/sentinels/outputs/quality-observer/quality-observer.log. The captured report confirms completion and duration, while explicitly saying that artifact existence, row coverage, and correctness cannot be verified from the event stream.
A complete live capture on 2026-09-27 ran all seven codons–normalize-aster, normalize-beacon, normalize-cedar, survey-and-extracts, validate-and-repair, reconcile, and award-brief–and matched the held-out oracle for the unified records and exception types. The taxonomy was earned over four live iterations: mismatches added low-confidence quarantine, page-split repair and logging, handwriting-to-notes routing, and prefix-split deterministic repair. A page-split repair joins a row across a page boundary and is logged as EX-07; a prefix-split repair recovers an exact leading part identifier and is a resolved mapping, so it needs no exception row.
Run the queue and catch omissions#
From the anchor hank's directory, run the commands below. The anchor run requires Bun, Git, Node.js >= 22.19.0, and Python 3 (the python3 interpreter); ANTHROPIC_API_KEY and BASETEN_API_KEY exported (a Claude Code login alone is not this recipe's authentication contract); and a non-root user with writable execution and output directories.
Run the preparation, validation, execution, oracle, and report checks in that order:
python3 ../verify.py prepare-data ../quote-template-unification task-data
bunx hankweave@0.11.0 hank.json task-data --validate
bunx hankweave@0.11.0 hank.json task-data --headless --start-new --execution exec --max-cost 9 --shim-idle-timeout 1800 --overwrite-output -o out
python3 ../verify.py anchor out
bun rigs/render-exceptions.ts out --check && bun rigs/render-award.ts out --check
Here, --max-cost 9 requests the anchor's configured $9 global ceiling; it is not a predicted bill.
The preparation command copies task inputs into task-data/, which the runtime exposes inside the execution at agentRoot/read_only_data_source/ as a symlink to the data directory (or a copy when the link cannot be made). Nothing enforces read-only there; the name is a convention the hank keeps. Preparation leaves the answer key, generator, and failure fixtures outside the agent's input directory. Reuse the prepared directory for repeat runs; preparation refuses to overwrite an existing destination. The <execution> placeholder in Check-it 4 is the directory named by --execution–exec in the command above.
--validate hashes the data source, validates configuration and paths, and runs the same per-model harness self-test as startup (SDK availability, credential presence, and the model-catalog preflight), so a missing key fails it with Self-test failed for N model(s). It runs no codons, and it does not prove key validity or provider health: the billable provider health checks run at runtime startup, separate from the tracked codon cost. Use --start-new for an independent execution. A repeat run into the same explicit --execution exec directory stops with "Directory already contains execution state" unless you add --force (which backs up .hankweave/ and wipes agentRoot/) or target a different execution directory.
Keep the held-out answer key separate. Do not point the corpus argument at a directory whose truth/ contents have been merged into anything a prompt reads. truth/expected.csv and truth/exceptions.csv are the answer key for the post-run comparison.
After the run, inspect the execution directory's agentRoot/ unless an output directory was set (-o, or outputDirectory in hankweave.json). With neither set there is no default output directory, and outputs stay in agentRoot/. Expect exception-ledger.json after validate-and-repair, unified-records.csv after reconcile, and award-brief.md plus exceptions.csv after award-brief. The bare quality-observer.log name resolves under the managed sentinel output path shown in Check-it 4, not beside the ledger.
Treat a clean exit as insufficient evidence. The silent-omission signature is an exit status of zero, a clean-looking award report, and no error while a ledger row is missing or a supplier-shaped hole is unnamed. The real catchers are the reconcile coverage rule and the deterministic verify.py oracle diff; the sentinel can report completion-event limitations but cannot establish artifact correctness.
Make completeness fail loudly. The reconcile task declares that a silently incomplete unified table is worse than no table and uses budget.onExceeded: "fail" for that stage. See the budget tutorial for budget-policy details.
The reconcile prompt excerpt below shows that coverage rule as written: every one of the 40 supplier-part combinations gets exactly one outcome, and a missing combination produces both an exception and an explicit UNRESOLVED row.
For each of the 8 suppliers × 5 BOM parts (40 combinations):
- **Exactly one record expected.** If `validated-records.json` has exactly one matching `(supplier_code, buyer_part_id)` record, carry it forward as-is.
- **Missing.** If no record matches, do not simply leave a gap in your output. Emit a new exception (`type: "MISSING_COVERAGE"`, next sequential `exception_id` continuing from wherever `exception-ledger.json` left off) explaining which supplier/part never resolved to any record, and still write a row for it in `unified-records.csv` with `status: "UNRESOLVED"`, `ranking_eligible: false`, and blank price/extension/source fields.
- **Duplicate.** If more than one record matches the same `(supplier_code, buyer_part_id)`, that's a conflict `validate-and-repair` should have caught but may not have — emit a `DUPLICATE_LINE` exception citing both `source_line_id`s, keep the first record encountered, and log the discard.
Now try the judgment drill on your own work. For each decision, choose one path:
- Automate: let code or a lookup decide it and put the result on a deterministic surface–a result repeated by the same code or lookup.
- Decide and log: when an agent can decide defensibly and a wrong answer is cheap to correct, decide and append an exception row.
- Route: when it is a human call that can wait, preserve the input in notes or quarantine.
- Escalate: when people may contest it, use a JC block with options, tentative resolution, confidence, a fail-closed result, and human sign-off.
Apply the drill to the anchor: the unknown template is quarantined (EX-01); dedup is decide-and-log (EX-02); NO_BID is an automated typed status (EX-03); expiry is JC-1 (EX-04); 'TBD' prices fail closed and remain null and ineligible (EX-05); low confidence is quarantine (EX-06); page-split repair is deterministic repair plus logging (EX-07); handwriting is routed to notes (EX-08); and prefix-split repair is automated from an exact leading part-id match with no exception row at all.
Check-it 5: State the failure signature: exit status zero, a clean award report, and a missing ledger row. Name both catchers: reconcile's MISSING_COVERAGE rule and the deterministic verify.py oracle diff–not the sentinel's completion report.
Use the captured hazard list as a starting point, not as a promise that another queue has the same types.