A hank is a sealed sequence of codon tasks; a codon is one bounded agent task in that sequence. Once a hank runs headless, nobody is watching it decide. Some decisions in a workload are safe for an agent to settle on its own, and some are not: a quote that expired before the award date, a handwritten discount attached to a bid, a supplier whose format nobody has seen before. If the agent quietly resolves those, the output looks complete and the mistake is invisible.
This page is a pattern for keeping those calls visible. The idea is simple: decide up front which choices the hank may make, record everything unresolved in durable files that the next codon or a human reviewer can inspect, and reserve a structured escalation format for the calls code cannot defensibly make. The running example is the quote-template-unification anchor hank, which reconciles quotes from eight suppliers against a five-part bill of materials and carries one planted judgment call (JC-1) through to human sign-off.
Map decisions before you freeze the hank#
The pattern starts before any codon runs. Before freezing a hank's task sequence, map the decisions in four columns:
- May make: decisions the hank may settle.
- Must surface: decisions it must show to a person.
- Who resolves: the person or role that settles each surfaced decision.
- Where durable: the file or other surface where the answer remains available.
That map places human judgment at both ends of the run. At authoring time, shape inputs interactively; then run the hank headless–that is, without interactive guidance. At the end, write a judgment-calls review record containing the options, tentative decision, confidence, and the consequence if the decision is wrong. Have a person review it and feed the outcome into the next run. The general pattern has no required filename, so choose one for your hank and document that choice.
The anchor applies this at its start judgment: hand-transcribe the request-for-quotation (RFQ) contract into reference/rfq-bom-rev-b.json from the corpus XLSX/PDF, then ship that JSON inside the hank. Do not ask a codon to parse the binary contract at runtime. Here, the anchor is the quote-template-unification sample hank used by the examples.
Use the anchor as a low-density example: it carries one planted judgment call (JC-1) across eight quoting suppliers. For the judgment-density fit boundary, see the fit guide; see the repair tutorial for the call's build walkthrough.
Stop continue from hiding unresolved work#
With the decision map in place, the first enforcement point is the exception queue. Make it a deliverable, not a log. We ask the agent to record unresolved material before moving past a problem, so a person can still find it. That is the intent of the prompt's Continue rule, not a guarantee that a run cannot be interrupted:
Continue must never mean silent omission: if a line, a part, or a whole quote doesn't cleanly resolve, it must appear in `exception-ledger.json` with a reason. It may never simply vanish from your output.
The final award-writing task also preserves one line per ledger exception. A row may be resolved, routed, or marked unresolved, but it must not be dropped, merged away, or summarized out of existence.
Give every standard ledger row these seven fields:
exception_id, type, supplier_code, buyer_part_id, detail, resolution, source_ref
Number exception_id as EX-01, EX-02, and so on in emission order. Pipe-join identifiers when one exception spans multiple parts or blocks. Keep the ledger append-only across codons: preserve earlier rows and append new ones.
Make coverage gaps visible. A missing supplier-by-part combination emits MISSING_COVERAGE and still receives an UNRESOLVED row in unified-records.csv; a missing supplier is a loud failure, not a silent skip.
Apply the same discipline to each hazard. The supplier names below–Aster, Cedar, Fjord, Granite, and Iris–are records in the quote-template-unification anchor; each illustrates a different kind of exception.
- Quarantine an unrecognized intake as
UNKNOWN_TEMPLATE, with zero extracted rows and an onboarding note. Granite's first-time XLSX is quarantined rather than force-fit into a template. - Fail closed on Fjord's
'TBD'prices: usestatus: UNRESOLVED, null prices, andranking_eligible: false. ANO_BIDline is not a zero-price quote. - Log mechanical conflicts. Aster's later
issued_atsubmission wins deduplication, while the one-cent stated-total drift remains aDEDUP_CONFLICTrow. - Route Iris's conditional handwritten offer as
NOTE_ROUTED, carrying the text verbatim. Leave the quoted line unchanged; the offer is a note for a person, not a price.
The captured ledger excerpt below shows one of these rows as emitted: Cedar's expired-validity exception, with all five part identifiers pipe-joined into buyer_part_id and a resolution that keeps the quote visible but ineligible.
exception_id,type,supplier_code,buyer_part_id,detail,resolution,source_ref
...
EX-02,EXPIRED_VALIDITY,CDR,NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A,Quote valid_until 2026-01-20 < award_decision_date 2026-02-01. Quote expired at moment of award.,Quote visible but ranking_eligible false on all lines; human review required,doc-cedar-cw-77
Check-it 1: After validate-and-repair, inspect exception-ledger.json. In the captured anchor output it is a JSON array of eight rows, EX-01 through EX-08, covering eight hazard types. EX-01 is Aster's dedup conflict, EX-02 is Cedar's expired-validity call across all five parts, and EX-05 is Cedar's NO_BID row; the judgment fields appear on EX-02 only.
Record the call code cannot automate#
Some exceptions are mechanical: a conflict to log, a gap to name. Others are genuine choices that people may reasonably contest, and those need more than a standard row. The shipped anchor uses a four-field judgment-call block in addition to the standard row, and omits those fields from every other row:
judgment_call: true
options: ["exclude", "show-flagged"]
tentative_resolution: "show-flagged"
confidence: 0.5
For JC-1, compare Cedar's valid_until (2026-01-20) with the RFQ award_decision_date (2026-02-01). Because the quote is expired at award time, set ranking_eligible: false on all five Cedar lines, regardless of each line's own status. Emit one EXPIRED_VALIDITY row for the whole quote, with the five part identifiers pipe-joined.
Keep the quote visible but unable to win ranking under the tentative show-flagged resolution. We leave the question open for a human override instead of hiding it in an automatic choice. Confidence 0.5 is deliberate uncertainty, not a confident automatic call. Review instructions may ask what happens if the choice is wrong; in this fixture's four-field shape, keep that consequence in detail and resolution rather than adding a fifth field.
The captures below show the call at three stages: the ledger row with its four judgment fields, the prompt step that instructs the agent to emit it, and the award-brief line that carries it to a reviewer.
...
{
"exception_id": "EX-02",
"type": "EXPIRED_VALIDITY",
"supplier_code": "CDR",
"buyer_part_id": "NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A",
"detail": "Quote valid_until 2026-01-20 < award_decision_date 2026-02-01. Quote expired at moment of award.",
"resolution": "Quote visible but ranking_eligible false on all lines; human review required",
"source_ref": "doc-cedar-cw-77",
"judgment_call": true,
"options": ["exclude", "show-flagged"],
"tentative_resolution": "show-flagged",
"confidence": 0.5
}
...
## Step 3 — Quote-level judgment call (JC-1)
Compare each quote's `quote_meta.valid_until` to the RFQ contract's `award_decision_date` (`2026-02-01`). Cedar's `valid_until` is `2026-01-20` — **before** the award date. The quote is expired at the moment of award.
This is a **planted, irresolvable judgment call**, not a bug to silently fix. Whether an expired-at-award quote should be excluded entirely or kept visible-but-ineligible is a human decision. Resolve it as follows and make the judgment visible, not buried:
- Set `ranking_eligible: false` on **every** line of the expired quote, regardless of that line's own per-line status.
- Emit exactly one exception of `type: "EXPIRED_VALIDITY"` for the whole quote (one row, `buyer_part_id` covering all 5 parts pipe-joined, e.g. `NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A`), with these additional fields beyond the standard ones:
- `judgment_call: true`
- `options: ["exclude", "show-flagged"]`
- `tentative_resolution: "show-flagged"` — the quote stays visible in the unified output and cannot win ranking, but a human reviewer can override this.
- `confidence: 0.5` (deliberately uncertain — this is not a confident automatic call).
- EX-02 (EXPIRED_VALIDITY, CDR): Cedar's quote validity period had closed prior to the award decision date, rendering the supplier ineligible for selection on all five parts despite formally submitting complete bids; this is a judgment call requiring human sign-off on whether to display an expired quote as a finalist option or exclude it from the competitive set. (source_ref: `doc-cedar-cw-77`).
Treat that captured assessment as the sign-off banner, not as the row-level eligibility source. The structured records distinguish Cedar's four QUOTED rows excluded by EX-02 from its NO_BID row for NC-1004-A, recorded as EX-05.
The held-out oracle encodes the same call under its own local row identifier, EX-01, and names it JC-1:
EX-01,EXPIRED_VALIDITY,CDR,NC-1001-A|NC-1002-A|NC-1003-B|NC-1004-A|NC-1005-A,CDR valid_until 2026-01-20 < award decision date 2026-02-01,"JC-1: planted irresolvable judgment call — exclude-vs-show-flagged is a human decision; quote stays visible, cannot win ranking",doc-cedar-cw-77
Row identifiers are local to each file; the call name is the stable label for the decision. Keep this answer key out of prompts and agent input directories.
Check-it 2: Confirm that the lossless exceptions.csv has the same header as the held-out answer key, that its expired-validity row carries the EXPIRED_VALIDITY identity and source reference, that the ledger row carries the four judgment fields, and that the award report calls for human sign-off.
Keep rendering from making new judgments#
Judgment belongs in the earlier stages and in human review. Make the final rendering task cite and format settled records. It must make no new judgment calls: every fact it states must already be settled in unified-records.csv or exception-ledger.json.
Require a [doc:block] citation for every dollar figure. Name Cedar's four ineligible QUOTED bids and cite EX-02; for NC-1004-A, show the NO_BID exception EX-05 rather than describing that row as an expired bid. Do not omit the quoted bids because they cannot rank. If no row for a part is ranking_eligible, say that no eligible winner exists.
Keep the capability boundary in the hank. validate-and-repair and reconcile use pi/baseten/deepseek-ai/DeepSeek-V4-Pro for judgment work. At the final stage, setup rigs render the financial section and serialize exceptions.csv; the haiku agent in award-brief writes only the qualitative assessment from settled records. We reserve the decision-making for the earlier stages and human review.
Check-it 3: Read the rendered outputs together. The captured unified-records.csv has 40 supplier-part data rows–eight quoting suppliers across five BOM parts, or 41 lines including the header–and the ledger has one row for each of the eight planted hazard types. The held-out oracle comparison matches those records and exception types.
Feed review into the next run#
Use a judgment-calls file as the post-run review surface for a headless hank. In this anchor, the EX-02 row in exception-ledger.json is that review record; its four-field shape keeps the judgment and its consequence durable without making that filename a general default. Have a person review each option, tentative decision, and confidence, then carry the outcome into the next run. The anchor evidence does not prescribe a filename or location for other hanks, so document your convention rather than relying on an implied default.
Keep the held-out answer key out of prompts. The run's corpus mount contains task inputs only; compare unified-records.csv and exceptions.csv with the answer key after the run, outside the agent's input scope.
Add a sentinel–an event observer that reports without editing–to the validation boundary. The anchor's quality-observer fires after validate-and-repair completes. It reports only status, duration, failure reason, and budget fields present in the completion event. It cannot inspect artifacts, establish row coverage, or infer missing files from absent file-update events. For sentinel configuration details, see the sentinel configuration reference.
The sentinel configuration below wires that boundary: the event expression passes the trigger events into the prompt, and the prompt text itself instructs the model to state what it cannot verify.
{
"id": "quality-observer",
"name": "Quality Observer",
"description": "Observes the validate-and-repair completion event. Reports only status and budget fields present in that event; cannot read or verify output artifacts.",
"trigger": {
"type": "event",
"on": [
"codon.completed"
],
"conditions": [
{
"operator": "equals",
"path": "codonId",
"value": "validate-and-repair"
}
]
},
"execution": {
"strategy": "immediate"
},
"model": "anthropic/claude-haiku-4-5",
"userPromptText": "The codon `validate-and-repair` completed. Here are the triggering completion events:\n\n<%= JSON.stringify(it.events, null, 1) %>\n\nSummarize only status, failure reasons and budget fields actually present. These completion events do not provide file contents or file.updated events. State explicitly that artifact existence, row coverage and correctness cannot be verified from this input. Never treat missing file-update events as proof of missing files. Observe and report only.",
"output": {
"format": "text",
"file": "quality-observer.log"
}
}
Check-it 4: After validate-and-repair, open the sentinel output at <execution>/.hankweave/sentinels/outputs/quality-observer/quality-observer.log. The captured report confirms completion and duration, while explicitly saying that artifact existence, row coverage, and correctness cannot be verified from the event stream.
A complete live capture on 2026-09-06 ran all seven codons–normalize-aster, normalize-beacon, normalize-cedar, survey-and-extracts, validate-and-repair, reconcile, and award-brief–and matched the held-out oracle for the unified records and exception types. The taxonomy was earned over four live iterations: mismatches added low-confidence quarantine, page-split repair and logging, handwriting-to-notes routing, and prefix-split deterministic repair. A page-split repair joins a row across a page boundary and is logged as EX-06; a prefix-split repair recovers an exact leading part identifier and is a resolved mapping, so it needs no exception row.
Run the queue and catch omissions#
From the anchor hank's directory, run the commands below. The anchor run requires Bun, ANTHROPIC_API_KEY and BASETEN_API_KEY exported (a Claude Code login alone is not this recipe's authentication contract), and a non-root user with writable execution and output directories.
Run the preparation, validation, execution, oracle, and report checks in that order:
python3 ../verify.py prepare-data ../quote-template-unification task-data
bunx hankweave@0.10.0 hank.json task-data --validate
bunx hankweave@0.10.0 hank.json task-data --headless --start-new --execution exec --max-cost 9 --shim-idle-timeout 1800 --overwrite-output -o out
python3 ../verify.py anchor out
bun rigs/render-exceptions.ts out --check && bun rigs/render-award.ts out --check
Here, --max-cost 9 requests the anchor's configured $9 global ceiling; it is not a predicted bill.
The preparation command copies task inputs into task-data/, which Hankweave mounts read-only as read_only_data_source/ inside the execution directory. It leaves the answer key, generator, and failure fixtures outside the agent's input directory. Reuse the prepared directory for repeat runs; preparation refuses to overwrite an existing destination. The <execution> placeholder in Check-it 4 is the directory named by --execution–exec in the command above.
--validate hashes inputs, validates configuration and paths, and performs setup and credential/catalog checks; it runs no codons or provider health checks and does not prove that credentials work. Runtime startup separately performs provider health checks, which can make model-generation calls and may be billable outside the tracked codon total. Use --start-new for an independent execution; omit it only when deliberately resuming an interrupted execution with the same data source.
Keep the held-out answer key separate. Do not point the corpus argument at a directory whose truth/ contents have been merged into anything a prompt reads. truth/expected.csv and truth/exceptions.csv are the answer key for the post-run comparison.
After the run, inspect the execution directory's agentRoot/ unless -o copied the outputs elsewhere. There is no default output directory. Expect exception-ledger.json after validate-and-repair, unified-records.csv after reconcile, and award-brief.md plus exceptions.csv after award-brief. The bare quality-observer.log name resolves under the managed sentinel output path shown in Check-it 4, not beside the ledger.
Treat a clean exit as insufficient evidence. The silent-omission signature is an exit status of zero, a clean-looking award report, and no error while a ledger row is missing or a supplier-shaped hole is unnamed. The real catchers are the reconcile coverage rule and the deterministic verify.py oracle diff; the sentinel can report completion-event limitations but cannot establish artifact correctness.
Make completeness fail loudly. The reconcile task declares that a silently incomplete unified table is worse than no table and uses budget.onExceeded: "fail" for that stage. See the budget tutorial for budget-policy details.
The reconcile prompt excerpt below shows that coverage rule as written: every one of the 40 supplier-part combinations gets exactly one outcome, and a missing combination produces both an exception and an explicit UNRESOLVED row.
For each of the 8 suppliers × 5 BOM parts (40 combinations):
- **Exactly one record expected.** If `validated-records.json` has exactly one matching `(supplier_code, buyer_part_id)` record, carry it forward as-is.
- **Missing.** If no record matches, do not simply leave a gap in your output. Emit a new exception (`type: "MISSING_COVERAGE"`, next sequential `exception_id` continuing from wherever `exception-ledger.json` left off) explaining which supplier/part never resolved to any record, and still write a row for it in `unified-records.csv` with `status: "UNRESOLVED"`, `ranking_eligible: false`, and blank price/extension/source fields.
- **Duplicate.** If more than one record matches the same `(supplier_code, buyer_part_id)`, that's a conflict `validate-and-repair` should have caught but may not have — emit a `DUPLICATE_LINE` exception citing both `source_line_id`s, keep the first record encountered, and log the discard.
Now try the judgment drill on your own work. For each decision, choose one path:
- Automate: let code or a lookup decide it and put the result on a deterministic surface–a result repeated by the same code or lookup.
- Decide and log: when an agent can decide defensibly and a wrong answer is cheap to correct, decide and append an exception row.
- Route: when it is a human call that can wait, preserve the input in notes or quarantine.
- Escalate: when people may contest it, use a JC block with options, tentative resolution, confidence, a fail-closed result, and human sign-off.
Apply the drill to the anchor: dedup is decide-and-log (EX-01); prefix-split repair is automated from an exact leading part-id match with no exception row; NO_BID is an automated typed status (EX-05); expiry is JC-1 (EX-02); page-split repair is deterministic repair plus logging (EX-06); 'TBD' prices fail closed and remain null and ineligible (EX-08); low confidence is quarantine (EX-03); handwriting is routed to notes (EX-04); and the unknown template is quarantined (EX-07).
Check-it 5: State the failure signature: exit status zero, a clean award report, and a missing ledger row. Name both catchers: reconcile's MISSING_COVERAGE rule and the deterministic verify.py oracle diff–not the sentinel's completion report.
Use the captured hazard list as a starting point, not as a promise that another queue has the same types.