Human-in-the-Loop Design: Building Better AI Automation Experiences

The review queue contains two hundred AI-prepared records. Every item has a green badge and an approve button. Reviewers can open the original document, but doing so takes several clicks. By the end of the afternoon, approval has become a routine gesture rather than a meaningful check.

Putting a person in a workflow does not automatically make the workflow dependable. Human-in-the-loop design must give that person the evidence, authority, time, and recovery options needed to make a real decision. Otherwise, the organisation has added a signature without adding effective oversight.

This article focuses on the design of review work in business automation. Imagine a team reviewing AI-extracted service applications before they enter an operational system. The same principles can inform document processing, editorial approval, support responses, and other tasks where assistance prepares work for human judgement.

Decide what the person contributes

A reviewer may verify factual extraction, resolve ambiguity, apply professional judgement, or authorise an external action. These are different responsibilities. A single approval control can conceal which one the organisation expects.

Write the review task explicitly. For an application, the reviewer might confirm identity matching and missing fields, while a separate manager authorises an exception. The interface should support those decisions separately when they require different knowledge or authority.

Do not ask a person to verify what they cannot observe. If a model makes a recommendation based on inaccessible sources, the reviewer cannot meaningfully assess it. Either provide appropriate evidence or narrow the decision.

Agree on the consequences of approval. Does it save a draft, submit a record, notify a customer, or release another automated step? Reviewers need to understand that boundary before the first live item reaches their queue.

Match review effort to the consequence of error

Not every suggestion needs the same interaction. Correcting a descriptive label may be lightweight, while approving a change with external effects requires more deliberate inspection. Treat the business consequence as a design input.

Identify which fields or actions matter most. A wrong contact identifier can be more consequential than an awkward sentence. Make critical information visible without requiring reviewers to inspect every low-impact detail at the same intensity.

Avoid declaring a case low risk solely because the model supplied a high confidence score. Use observed performance, task context, and enforceable business conditions to define review policy.

Revisit the policy when the workflow changes. A classifier used only for internal routing has a different effect when its output begins triggering customer messages automatically. The review interface should evolve with that authority.

Put source evidence beside the proposal

The fastest path to useful review is often a direct comparison. Show the original value or passage beside the proposed field, with clear highlighting of the relevant source region when feasible.

For a document workflow, reviewers should not need to search a long PDF for every attribute. Preserve source page and location references during extraction so the interface can open the supporting material directly.

Distinguish unsupported values from values that merely use a normalised representation. A converted unit and a guessed date need different attention. Present the transformation or uncertainty in a way the reviewer can assess.

Keep the evidence version aligned with the proposal. If a source document changes, an old approval should not be interpreted as a review of the new material. Store the relationship explicitly in the workflow.

Make the important differences visible

When reviewing an update, show what changed. Requiring users to reread a full record invites missed edits and wastes attention. A field comparison or revision diff can make the decision much more manageable.

Include related changes that affect meaning. A revised recipient, date, amount, or publication destination may be more important than the main body text. Do not hide those fields in a secondary panel while presenting a large approve button.

Use clear text in addition to colour. Reviewers may have colour-vision differences, use monochrome displays, or rely on assistive technology. Status and uncertainty should remain understandable without visual styling alone.

Avoid decorating every item as urgent. If everything is highlighted, nothing is prioritised. Reserve strong visual emphasis for conditions that actually change the review decision or require immediate attention.

Give correction the same attention as approval

A reviewer should be able to fix one field without discarding a useful draft. Provide local edits, an unresolved state, and a way to request additional information. Rejection should not be the only alternative to accepting everything.

Preserve corrections during later processing. A retry or regeneration must not silently replace a human-edited value. Store the edited proposal version and make subsequent differences reviewable.

Separate feedback for product improvement from the action needed to finish the current case. A category such as incorrect extraction may help the development team, but the reviewer still needs to enter the correct value and move the work forward.

Explain whether a correction changes future behaviour. A one-time edit is not necessarily a new rule or saved preference. If the organisation wants reusable mappings, provide an explicit governed path to create them.

Design a queue that supports judgement

The queue should show enough context to choose the next item: age, task type, unresolved conditions, required expertise, and relevant deadline. Sort by operational need rather than by whatever order model responses happened to finish.

Group similar work when it reduces context switching, but avoid batches so large that review becomes mechanical. A set of comparable fields can be efficient; a mixed set of unrelated consequential actions may need individual decisions.

Show ownership and handoff status. Reviewers should know whether another person is already working on an item and whether their decision will conflict with a concurrent edit.

Provide deliberate deferral with a reason. Some cases cannot be resolved immediately because evidence is missing. Keeping them in the same apparent state as untouched work makes workload and escalation harder to manage.

Make approval specific and enforceable

Bind approval to a proposal identifier, version, actor, and operation. The application should verify that these still match when it executes the action. Conversation text saying approved is not a sufficient record.

If material content changes, require the appropriate renewed decision. The organisation may define minor edits that do not reset approval, but those rules should be explicit and implemented consistently.

Recheck permission and current business state at execution. The reviewer may have lost the relevant role, or another process may have changed the record. Approval of a stale proposal should not overwrite a newer valid decision.

Show a receipt after execution based on the actual system result. Approved, submitted, and completed are separate states. The interface should not imply that a destination accepted an action merely because a reviewer clicked a button.

Avoid turning explanation into persuasion

An AI-generated rationale can help identify evidence, but it can also sound convincing when the proposal is wrong. Design the review around inspectable facts and conditions rather than a persuasive narrative urging acceptance.

Ask for concise supporting references and unresolved assumptions. A long explanation that repeats the proposed conclusion may consume attention without improving verification.

Where application rules make a decision, show their recorded result. Where the model makes a suggestion, label it as such. Reviewers should be able to distinguish enforced policy from generated interpretation.

Microsoft Research's human–AI interaction guidelines address control, correction, and communication of capability. Applying those principles to a review queue means making disagreement and repair practical, not merely adding a disclaimer beside the output.

Account for fatigue and workload

Review quality depends on the conditions under which people work. A design tested on five carefully chosen examples may behave differently when staff face a large backlog or frequent interruptions.

Measure time per accepted item, correction effort, queue age, and errors missed during review. Inspect variation across case types rather than relying only on an overall average.

Do not solve overload by quietly reducing the evidence shown. Consider narrower eligibility, better grouping, improved source presentation, or additional staffing. The workload should inform the automation boundary.

Make it safe for reviewers to flag an unsuitable case. If they are measured only on throughput, they may feel pressure to accept ambiguous output. Align operating measures with correct resolution and appropriate escalation.

Build for keyboard and assistive-technology use

Reviewers often perform repetitive work where efficient keyboard access matters. Provide predictable focus order, meaningful labels, and clear shortcuts where appropriate. Avoid forcing precision pointer movements for every field decision.

Dynamic updates should communicate state without overwhelming users. Announcing every streamed token or moving focus unexpectedly can make the interface difficult to use. A stable completed proposal is often more suitable for careful review.

Ensure source comparisons remain usable at increased zoom and on smaller displays. A side-by-side layout may need an alternative view that preserves the relationship between value and evidence.

Test the actual review task with assistive technology rather than checking only isolated controls. The user must be able to inspect evidence, correct a value, understand consequences, and complete the decision as one coherent journey.

Handle cancellation and partial execution honestly

A reviewer may withdraw approval before execution or discover a mistake after part of a workflow completes. Define what the system can stop and what requires a separate recovery action.

Do not label a sent message undone simply because the local task was cancelled. A reversible record edit and an externally delivered communication have different recovery boundaries.

Show completed steps, pending steps, and uncertain outcomes. If a destination timed out after a request, the system may need reconciliation before another attempt. The reviewer should not be asked to guess whether retrying will duplicate the action.

Provide a supported escalation path with the operation reference and evidence. A person dealing with recovery needs more than the original draft; they need the confirmed execution history.

Evaluate whether review catches meaningful errors

Use test cases with realistic mistakes, missing evidence, and conflicting records. Observe whether reviewers notice the problem without being told where it is. A prototype that always returns correct output cannot test oversight.

Measure both mistaken acceptance and unnecessary rejection. A design that causes users to distrust every result may eliminate the intended benefit. The aim is accurate decisions with proportionate effort.

Include interrupted sessions and batch work. Ask participants to resume an item after a delay and explain what has been approved or completed. This reveals whether workflow state is understandable beyond the initial interaction.

Compare review with the existing process. If assisted preparation saves time but verification becomes substantially harder, the design may need a narrower output, clearer evidence, or a different placement for AI.

Keep humans involved where they have actual authority

A nominal reviewer who cannot change the result or stop the action provides little meaningful control. Ensure the role can reject, correct, request information, and escalate within the agreed process.

Separate specialist judgement from administrative checking. A data-entry reviewer may verify fields but lack authority to approve an exceptional business decision. Route that decision to the appropriate role instead of relying on a general approve button.

Document responsibility for unresolved cases. A queue can become a place where ambiguity is stored indefinitely if nobody owns the next step. Assign routes based on the evidence or judgement missing.

Review these responsibilities with staff before launch. The system should support their expertise rather than use their presence as an explanation for risks they cannot realistically control.

## Test what happens when reviewers disagree

Two qualified reviewers may reach different conclusions because the source is ambiguous or the policy leaves room for judgement. The workflow should preserve the disagreement and route it appropriately rather than treating one click as proof that the issue is objectively resolved.

Allow a reviewer to identify the point of uncertainty and request a specialist decision. Record the eventual resolution so similar cases can be handled more consistently where the policy allows it.

Do not turn every disagreement into automatic model retraining. Some corrections reflect an exceptional decision, while others reveal a reusable rule. A domain owner should distinguish those cases before changing broader behaviour.

Review disagreement patterns during the pilot. They can expose unclear instructions, overlapping responsibilities, or missing evidence that would also affect a fully manual process.

Make audit history readable during a handoff

A colleague taking over a case should see which fields were checked, what changed, who approved the current version, and whether execution completed. Present that history as business events rather than an unfiltered technical log.

Keep the original proposal and the reviewed revision distinguishable. This supports investigation and helps the team understand whether an error came from generation, review, or execution.

Use role-appropriate detail. An operations reviewer may need the source and decision history, while a support engineer needs an operation identifier. Both views should describe the same confirmed state.

Test a handoff after an interruption to verify that a new reviewer can continue without repeating completed checks or relying on stale approval.

Start with one decision worth designing well

Choose a review point where the output can be checked and the result matters. Build the complete journey from source evidence through correction, approval, execution, and recovery before expanding the volume.

Use early observations to refine the queue and eligibility rules. Some case types may be suitable for efficient assisted review, while others should remain with the existing process until the evidence or interface improves.

The success criterion is not how often a person clicks approve. It is whether the person can make an informed decision, complete useful work, and recover when something goes wrong. Design for that outcome and human involvement becomes a working part of the system rather than a reassuring label.

---