Architecture

Human-in-the-Loop AI: Design the Review, Not Just the Approval Button

Orisdale Editorial Team · Published · 8 min read

Design the review packet, the states, and the roles before adding an approval button. Accept, reject, and escalate are requirements for that design, not features this site has deployed.

The button is the last control, not the design

Human-in-the-loop is often drawn as a button under a paragraph. The click can be real while the design is missing: which contract version produced the number, which evidence was linked, which sentence is a cause, and what approval will mean tomorrow.

Orisdale's public standard is narrower. A person confirms a consequential finding. The agent may explain governed fields. It may not invent a cause. That standard only works if the review is a packet with states, not a gesture. The agent runtime and the architecture both leave room for that confirmation. This note says what the confirmation has to contain. Accept, reject, and escalate, as used here, are design requirements. They are not a workflow product running on this site, and they are not a claim that a customer review queue exists.

BlueprintReview states and their branches. Not a deployed review product.
ProposedA packet exists. Not routed yet.
Awaiting reviewWith a named reviewer and contract version.
  1. AcceptedReviewer records which claims were accepted.
  2. RejectedReviewer records the reason. The number can still stand.
  3. Needs more evidenceA reference was unusable, or an input was missing.
Needs more evidence returns to Awaiting review once the missing reference is added. It is not a move to Accepted.
Escalate is a separate path from Awaiting review, to the rule owner or a second reviewer. It is not a softer reject, and Accepted does not lead to Rejected.

Risk decides who has to look

Not every row deserves the same review. A design that sends every sentence to the same queue risks reviewer fatigue and sampling instead of attention, and a design that sends nothing treats the model as the approver. Risk is how the split should be chosen. Risk here means the consequence of being wrong, plus the chance the packet is incomplete. It is not a score this article computes.

This is a proposed routing policy, not a feature of the published proof. In the finance proof, Material means an absolute variance of at least 5,000.00 USD or at least 10 percent, and Management attention is the next band: not Material, and at least 1,000.00 USD or at least 5 percent. Those category labels are demonstration rules, not a customer's policy, and the proof does not route any of them to a review queue by materiality — the calculation's own reviewRequired field is a separate, narrower flag that is only true for the zero-budget case ("Needs human review"), and it is false on every published row, including the Material EX-FIELD-OPS result. A materiality-based routing policy of the kind described in this section is a design proposal this note is making. It is not something the proof demonstrates.

The routing rule has to be owned and visible. If "high risk" is only a feeling in the prompt, the queue drifts whenever the prompt is edited. The trust layer note puts human approval outside the model. The button can record a decision. It cannot decide which cases deserve one.

Reviewer, rule owner, and operator

Three roles get merged in demos and must be split in a design.

The reviewer decides a case under the contract version in the packet. The synthetic proof records no such act. Causal confirmation stays "not recorded."

The rule owner changes the contract: a threshold, a grain, or a rejection rule. Agreeing with one narrative is not that power. A new materiality test belongs on a new version, as a design requirement in the business logic contract note, not as running software.

The operator keeps the runtime and the log. They do not edit a threshold to clear a queue, and they do not accept a case because the reviewer is out. If one person wears both hats, the log has to say which one. "Approved" alone does not.

What the packet contains

If a field is missing from the packet, the reviewer is left to supply it from memory or from the fluency of the paragraph. Both are bad sources.

Packet fieldWhat the reviewer is being shown
Decision and grainThe question this case answers, and the unit it answers for
Contract versionThe rule that produced the result, so a later reader can tell version 1.0.0 from a successor
Governed resultThe number, direction, and category the rule produced, separate from any sentence about cause
Evidence referencesRecords the claim named, with their identifiers. A search hit that merely shares a period is not this field
Claim splitWhich sentences are numeric and which are causal
Missing dataWhat stayed blank, including a percentage that the rule refused to invent
FreshnessWhen the inputs were current, and which close or period they belong to
Proposed stateThe state the case is in now, and the action the reviewer is being asked to take

The finance worked example is the smallest public packet of this kind. For EX-FIELD-OPS in 2026-09, the governed result is variance 15,000.00 USD, over budget, Material, 15.0000 percent. Claim C3 names evidence E1, an equipment-rental line item in that department's expense detail. The status is "evidence linked — human verification required." The packet can show that. It cannot show a recorded confirmation, because the calculation does not collect one.

The states a case can be in

Five states stop a binary button from hiding the work. They are a design vocabulary, not statuses in an Orisdale application.

Proposed. A packet exists and has not been routed. Proposed is not a silent yes.

Awaiting review. The case is with a named reviewer under a named contract version. This article does not report how long that takes.

Accepted. The reviewer records which claims were accepted. Accepting the number is not accepting the cause unless the cause was part of that record.

Rejected. The reviewer records the reason. A wrong cause can be rejected while the Material classification stands. Rejection does not delete the governed number or rewrite the paragraph until it passes.

Needs more evidence. A named reference was unusable, an input was missing, or a hypothesis cannot be told from a fact. Letting the model guess is not a move to Accepted.

The finance contract uses different labels for a different job: "Pending human review," "Needs human review (rule-based)," and "Rejected input." Those come from the calculation, including zero budgets and bad inputs. "Needs human review" can open a case in Awaiting review. It is not itself a completed review.

Accept, reject, and escalate

These actions are design requirements. They are not features on this website.

Accept records the reviewer, the time, the contract version, and the claims covered. An acceptance that omits the version cannot be read after the rule changes.

Reject records the reason, and whether the number was refused or only the narrative. "Looks wrong" is a weak record. A field, a missing reference, or a contradicted cause is a usable one. The proof's checker will not write that reason.

Escalate is not a softer reject. One path goes to the rule owner when the dispute is about version 1.0.0 itself. Another goes to a second reviewer when the case is outside the first reviewer's authority. The operator may route the escalation and may not decide it.

How review fails without a bad model

Stale context. The packet was built on an earlier extract. The reviewer accepts a September sentence that no longer matches the files. If freshness is missing, the packet is unauditable.

Changed rules. Commentary under version 1.0.0 is not silently re-labeled when a threshold moves. The case returns to Proposed or Awaiting review against the new version. Accepting the old packet did not pre-approve the new rule.

Overload. If every row is Awaiting review, reviewers sample by fatigue. The design response is risk-based routing: Material cases and unresolved evidence ahead of within-tolerance restatements, which stay available rather than hidden. No queue depth or hours saved is stated here. Those numbers are not known.

Automation bias. A paragraph that repeats a correct number is easy to accept in full, including a cause that was never checked. In the finance proof, numeric agreement can be exact while causal confirmation stays "not recorded." The counter is the claim split: the cause has its own field and its own status. No prevalence figure is stated, because this note is not citing a measured rate.

Measures worth defining, without results to report

These are definitions only. None of them has a published result.

  • The share of causal claims that reach a recorded confirmation instead of remaining "not recorded."
  • The share of packets returned to Needs more evidence because a reference or an input was missing.
  • Whether an accepted number still matches the official result for that contract version. The synthetic proof checks exact equality for its own rows. That is not an organizational accuracy rate.
  • Elapsed time from Awaiting review to a recorded state, split by risk band.
  • How many escalations were actually rule changes.

A demo that quotes a percentage against this list is reporting something this site does not have. The worked example is the packet that stops where the evidence stops.

Where to take a real decision

Bring a decision that already has a rule owner and a reviewer, even if they are the same team today. The contact page is where to request a strategy session. The session can separate the case review from the rule change. It cannot quote a cycle time or a deflection rate, because Orisdale is not publishing those results.

Read the trust boundary for where approval sits relative to the model, the finance proof for a packet with real synthetic fields, and the architecture for the layers that have to be in place before a reviewer is asked to trust the screen.

Discuss how this applies to your environment