Tianyang Liu

Essay, 8 September 2026English / 中文

The Human Is Part of the Agent

Rethinking the Human’s Place in the Age of Agents

39 min read Agent Evaluation Written by GPT 5.6 Sol, Claude Opus 5, and Gemini 4 Argon

How this post was made

This blog was researched by GPT 5.6 Sol and written by Claude Opus 5. GPT 5.6 Sol also designed and implemented the figures, with only light human involvement in the final polish of both the writing and visuals.

TL;DR

Models keep climbing on benchmarks, yet in everyday work an agent can still occasionally feel “stupid.” Several distinct problems sit inside that frustration. Lapses in execution control, such as forgetting a constraint, changing files without checking, or failing to stop, have clear engineering levers in training and harness design. Real delegation also runs into a different difficulty: what to build and what counts as a good result often become clear only after someone sees a concrete attempt. Benchmarks fix the task, environment, and grading rule in advance, whereas real work requires forming the goal, verifying that the result is worth adopting, and deciding who can authorize actions and bear responsibility.

As agents take over more execution and routine checks, human effort changes location. Judging whether delegation saves time requires counting everyone involved: the person who initiates the task, the reviewer who checks the change, and the maintainer who inherits it later. Predefined permission scopes and automated tests can replace repetitive step-by-step approvals and catch surface breakage before a person looks. What reaches a human reviewer is then work that already passes automated checks, concentrating human effort on judgments machines cannot settle: whether the result matches what was actually wanted, whether the implementation is maintainable enough to keep, and whether the team understands the change well enough to own it when something breaks.

Contents, 6 sections
  1. 1What, Exactly, Feels Stupid?
  2. 2What Benchmarks Fix for the Agent
  3. 3We May Have Been Evaluating a User Who Does Not Exist
  4. 4Where the Cost of Delegation Goes
  5. 5Which Loop Should the Human Be On?
  6. 6The Human Is Part of the Agent

Over the past few years, the field has tracked model capability by building ever harder benchmarks. When MMLU stopped separating frontier models, researchers introduced GPQA; as competition mathematics approached saturation, evaluations moved to harder AIME and IMO problems and then to Humanity’s Last Exam; coding benchmarks progressed from HumanEval to SWE-bench, Terminal-Bench, and software engineering tasks that unfold over much longer time horizons. Keeping pace has grown difficult. The Stanford 2026 AI Index notes that some benchmarks intended to last for years began approaching saturation within months, and on Humanity’s Last Exam, which was designed specifically for frontier models and weighted toward questions that challenge human experts, the top score rose by roughly 30 percentage points in a single year (Stanford Institute for Human-Centered Artificial Intelligence, 2026). Extending the ruler makes sense for what these suites measure: a set of repeatable, objectively scored problems offers a tractable proxy for how capable a model has become.

As those suites grow harder, model scores are reaching familiar human reference points such as olympiad medals, domain expertise, and open research problems. In July 2025, OpenAI’s general-purpose reasoning model scored 35 out of 42 on the International Mathematical Olympiad, reaching gold-medal level. By February 2026, experts judged at least five of ten proof attempts in First Proof likely to be correct, including problems that had remained open for years before their authors found solutions (OpenAI, 2026). In August 2026, ten announced results in mathematics and theoretical computer science each solved or substantially advanced a longstanding open problem (OpenAI, 2026). Those ten results involved multiple attempts, human selection, and expert feedback, so treating every one as an independent model achievement would be inaccurate, even as the direction of capability growth remains clear. In cybersecurity, the same capability curve carries immediate operational consequences. Mythos Preview can discover and exploit zero-day vulnerabilities in major operating systems and browsers, finding many previously unknown vulnerabilities autonomously after an initial prompt (Anthropic, 2026). Because that capability is asymmetric between offense and defense, programs such as Project Glasswing give defenders earlier access (Anthropic, 2026).

Following this curve alone suggests that agents have become extraordinarily smart. They solve mathematical problems beyond the reach of most people, contribute at research frontiers accessible to only a small number of domain experts, and outperform human experts on some highly specialized tasks. On many frontier tasks, an ordinary person has moved from asking whether they would do better than the model to wondering whether they can even follow what it is doing. Yet daily work with Claude Code, Codex, or any comparably capable agent produces a more complicated impression. The model is unquestionably powerful, sometimes absurdly so, and yet on tasks that appear far easier than olympiad mathematics or vulnerability research it can still occasionally feel strangely stupid. A task may finish, the code may run, and the output may contain no identifiable factual error, while something still feels wrong. Switching models, changing the interaction style, or adding a few turns of back-and-forth can change how pleasant the agent is to use in ways that benchmark accuracy does not capture.

That gap arises because the single word “smart” is being asked to evaluate five different system properties. A benchmark primarily measures problem-solving capability: given a fully specified problem, how difficult a problem can the solver handle? Everyday delegation exposes four additional questions. Agentic control asks whether the agent can apply its capability reliably across a long trajectory. Task formation asks whether the system can continue forming and revising the task through interaction. Assurance asks whether there is sufficiently independent evidence to justify adopting the result. Governance asks who authorizes an action and how responsibility is assigned when something goes wrong. All five can overlap in a single failure, and they improve at different rates (Figure 1). When all five are compressed into one judgment about whether the agent is smart, benchmark scores can keep rising while day-to-day usability fails to improve at the same pace.

What smart compresses A governance boundary surrounds the full delegation system. Task formation enters from the left and assurance extends to the right. Agentic control surrounds problem-solving capability and extends beyond a rectangular solver benchmark window. Within that window, capability is measured and only part of control is exposed. The task and verifier are supplied at the two benchmark boundaries, and the environment is fixed. What “smart” compresses Governance boundary authority · accountability Task formation goal · context · preference Assurance tests · review · audit Agentic control partly exposed task supplied verifier supplied Problem-solving capability measured solver benchmark environment fixed
Figure 1. What “smart” compresses. A solver benchmark receives a supplied task, fixed environment and supplied verifier; it measures problem-solving capability and exposes only the intersecting part of agentic control. Task formation, assurance and governance belong to the larger delegation system. The boundaries are functional, not quantitative.

What, Exactly, Feels Stupid?

When people complain about an agent in daily work, they rarely mean that the model lacks reasoning ability. Typical complaints sound much more specific: “You knew that constraint, so why did you violate it again?”, “You started changing things without confirming anything,” “I just changed that, didn’t you notice?”, “You said it was done without verifying it,” or “You made a mess and can’t roll it back.” Every complaint assumes that the model was capable of doing the right thing; the failure lies in how it managed that capability across an unfolding trajectory. Grouping these complaints by the system function they implicate shows that they arise at different layers and require different remedies:

What the user feels A more precise description Primary system function
“You knew that constraint. Why did you violate it again?” constraint persistence agentic control
“You started changing things without confirming anything.” evidence threshold agentic control
“There was nothing to change, yet you changed it anyway.” action bias agentic control
“You forgot your own plan halfway through.” plan adherence agentic control
“You said it was done without verifying it.” self-report calibration agentic control
“Your changes keep expanding and you won’t stop.” scope control agentic control
“I just changed that. Didn’t you notice?” shared state reconciliation agentic control
“You made a mess and can’t roll it back.” recoverability agentic control
“You wrote the tests and then used them to validate your own work.” non-independent verification assurance
“You would rather guess than ask me.” help-seeking calibration task formation
“I had to see the result before I knew I didn’t want it this way.” objective reconstruction task formation
“You had no authority to make that decision for me.” authority governance

The first eight rows mostly concern agentic control, which model training and the harness must address together. A constraint or plan may drop out of the context after dozens of steps, or it may remain present without being followed; most failures involve some combination of both, so assigning every row exclusively to one side is rarely clean. The ninth row crosses into assurance. Treating “you wrote the tests and then used them to validate your own work” as an assurance problem is my own judgment rather than a claim directly established in the literature: if the agent has misunderstood the requirement, its tests may drift in the same direction as its implementation, so a green test run provides no independent evidence. The tenth and eleventh rows belong to task formation, covering information that has not yet been obtained and a judgment that has not yet been formed. The final row belongs to governance, which determines which actions require approval and from whom. Although real failures often span multiple rows, the table identifies the first problem each complaint reveals.

Most of the empirical evidence below comes from coding agents. Software engineering provides executable tests, git, diffs, CI, sandboxes, and rollback, which make failures unusually visible, localizable, and countable compared with legal, medical, financial, or organizational decisions, where correctness may itself be ambiguous and reversal difficult. The coding literature is useful for identifying mechanisms, though its measured proportions should not be generalized directly to every industry.

Observational logs show how common these failures are in practice. An analysis of 20,574 coding-agent sessions across 1,639 repositories identified seven recurring forms of misalignment spanning project understanding, intent interpretation, rule adherence, action boundaries, code execution, and progress reporting. In 90.5% of episodes, the failure imposed a cost on effort or trust. Because most sessions did not confirm the final outcome in the visible conversation, resolution could be observed in only 9.33% of episodes (1,504 cases); among those, 91.49% required explicit user correction (Tang et al., 2026). As the overall problem rate declined in the dataset, constraint violations and inaccurate self-reports made up a larger share of what remained. Observational data cannot establish that stronger models caused this shift, though it suggests that the remaining problems increasingly fall outside a simple inability to perform the task.

Controlled experiments reproduce two of these patterns directly. FixedBench tests whether an agent knows when to stop: all 200 of its tasks come from issues that had already been fixed, so the correct action is to recognize that fact and leave the code unchanged. Across five recent models and four harnesses, agents still proposed unnecessary changes in 35% to 65% of cases. Requiring them to reproduce the problem before patching reduced some over-action while creating under-action on issues that had been only partially fixed (Gloaguen et al., 2026). FixedBench does not establish why agents are so inclined to act; one plausible explanation is that in most training and evaluation signals they have seen, doing nothing is rarely counted as success. A second benchmark, SWE-Touch, tests what happens when the environment changes while the agent is working. In real development, users read, edit, and sometimes revert code during an agent run, so the repository state is jointly produced. Injecting simulated user edits relevant to the task and capable of changing the completion path into SWE-bench Verified reduced the average resolve rate by 7.7 percentage points, with trajectory analysis showing agents retaining code that conflicted with user edits or overwriting those edits without rechecking the workspace or running targeted tests (Tan et al., 2026). Standard SWE-bench never exposes this behavior because nobody else modifies its environment.

Final pass or fail labels also reveal little about where an agent first went off course. Human annotation of 115 failed trajectories across structured API workflows, incident management, and open-ended web and file tasks identified nine failure categories, including plan-adherence failure, invalid invocation, misreading tool output, intent-plan misalignment, and underspecified user intent, while attempting to locate the first unrecoverable critical step in each trajectory (Barke et al., 2026). The same failure modes recur across long trajectories even when the task has nothing to do with writing code.

Calling these breakdowns “the model cannot do the task” has become increasingly misleading. The model is usually capable of completing each local step; the breakdown happens over dozens or hundreds of steps, when it must still honor original constraints, maintain current state, gather evidence before acting when evidence is weak, control scope, know when to stop, verify the outcome, report status honestly, and recover after a mistake. This layer of agentic control is distinct from the difficulty of the problems a model can solve, and it is the target of many complaints that begin, “You clearly know how, so why did you do that?” I expect this layer to improve substantially because each component offers concrete engineering levers: persistent constraints, refreshed workspace state, pre-action evidence thresholds, automated verification, checkpoints, and rollback can all be addressed through the harness, context management, and training incentives. That remains a prediction without long-term evidence, and FixedBench’s under-action trade-off shows that reducing one control failure can produce another. A more defensible expectation is that a meaningful share of today’s perceived stupidity will decline as systems improve. That leaves a natural question: if control failures are so common in real use, why are they nearly invisible on benchmarks?

What Benchmarks Fix for the Agent

Benchmarks largely miss these problems by design, and that design is appropriate for what they measure. SWE-bench gives an agent a real GitHub repository, an issue description, and a code environment, then uses tests to determine whether the resulting patch works (Jimenez et al., 2023). SWE-Lancer extends the same idea to more than 1,400 tasks drawn from real freelance work and maps them to roughly one million dollars in actual payouts (Miserendino et al., 2025). Both answer a solver question: given a written task and a fixed success criterion, how difficult a problem can the model solve? Meaningful measurement requires the task, environment, and verifier to remain fixed. In real delegation, a larger system must continually supply and maintain those conditions across the agent, harness, user, CI pipeline, and organizational process. The task may be reinterpreted, other people may modify the environment concurrently, success may depend on preferences the user has not articulated, and validation may come from an independent checker or reviewer. Because a benchmark fixes these variables on behalf of the whole system for the duration of a run, a component capability score cannot by itself predict how the same model will perform in a real delegation.

Preparing those clean benchmark conditions takes substantial human work beforehand, and it is expensive. DeepSWE designed 113 long-horizon software engineering tasks from scratch across 91 real repositories and hand-wrote a behavior-oriented verifier for each task, accepting implementations the researchers had never seen so long as they satisfied the requested functionality. Independent LLM judges disagreed with those verifiers only 1.4% of the time, compared with 32.4% disagreement on tests inherited from SWE-Bench Pro (Huang et al., 2026). The scarce human judgment in DeepSWE sits upstream of solving the 113 tasks: deciding which tasks are worth measuring, which complexities of a real environment should remain, and what an unfamiliar solution must satisfy to count as success.

This shift upstream began well before modern agent evaluation. In early RLHF, labelers wrote ideal responses to user prompts for supervised fine-tuning, ranked multiple model-generated candidates, trained a reward model on those rankings, and used it to optimize the policy (Ouyang et al., 2022), spending most human effort on individual samples. Models later absorbed much of that sample-level work. In 2023, ChatGPT outperformed crowdworkers on four of five text-annotation task categories, achieved higher inter-coder agreement, and cost roughly one twentieth as much per item as MTurk (Gilardi et al., 2023), while GPT-4 acting as a judge agreed with human preferences more than 80% of the time, with systematic costs such as position and verbosity bias (Zheng et al., 2023). Because the work being absorbed consisted of bounded judgments attached to single samples, human effort could move toward judgments with broader coverage. Constitutional AI made this shift explicit: humans write a set of principles, and the model uses the constitution to critique a response, identify violations, revise it, and compare responses to generate preference data for reinforcement learning (Bai & others, 2022). Instead of judging every response individually, a human can write one principle that constrains an entire class of future cases.

Across these different research threads, a common dimension emerges: how much future behavior a single human judgment can constrain, which I will call supervision leverage (Figure 2):

Supervision leverage A horizontal scale has four positions. An answer or label constrains one sample. A rule constrains a family of outputs. A rubric or verifier defines a set containing multiple visibly different valid solution paths. A task and environment define the boundary of the whole problem space. The examples grow in scope from left to right without implying a quantitative scale. Supervision leverage broader answer / label rule rubric / verifier task / environment one sample a class of outputs many valid solutions the problem space
Figure 2. Supervision leverage. A sample-level label constrains one answer; a rule constrains a family of outputs; a verifier defines a set of acceptable solutions; task and environment design set the problem space in which all of them operate. The widening marks judgment coverage, not measured labor or time.

Toward the left of the spectrum, the human produces or grades individual answers for the model; toward the right, the human defines the problem space in which the model operates and what success means inside it. A label affects one sample, a principle constrains a class of outputs, a verifier defines a set of acceptable solutions, and the task and environment establish the boundary of the entire problem space. This spectrum describes judgment coverage; by itself, it says nothing about whether total human hours rise or fall. It supports a narrower conclusion: some sample-level labor is declining while judgment moves into principles, verifiers, task design, and environment design. Those new positions often fall outside the scoring boundary: a SWE-bench score does not account for the judgment spent designing its environment, just as a successful delegation does not account for all the context a user supplied to make it work.

This progression still rests on an underlying assumption: that the task, rubric, and objective can be fully specified before the agent begins. Benchmarks turn that assumption into a design condition, which is how they measure solver capability reliably. User-facing agents rarely receive such clean conditions.

We May Have Been Evaluating a User Who Does Not Exist

Standard evaluations contain an unusually cooperative user: someone who states the full requirement at once, provides all relevant context, defines the success criterion upfront, and holds the same judgment throughout the task. They speak once at the beginning, and tests, a verifier, or a judge determine correctness afterward. Isolating problem-solving capability requires this kind of user, yet real work rarely provides one.

Consider an engineer who says, “This feature has become too complicated. Simplify it.” At the start, they may be unable to state how simple is simple enough. After seeing the first revision, they realize that their actual concern is whether a new teammate can understand the code, and that some apparently redundant branches encode lessons from earlier production incidents. The original request was valid, yet it needed a concrete alternative before it could become specific enough. A behavioral analysis of 11,579 AI-assisted IDE sessions found the same pattern: developers rarely state everything at once, tightening the objective instead through a sequence of short prompts and changing direction after seeing intermediate results (Tang et al., 2026).

Three different processes sit inside that back-and-forth. Sometimes the user knows what matters from the start and the agent simply lacks that information; the agent needs to ask, which we can call reveal. Sometimes the user has only a rough direction and needs to see an implementation and its trade-offs before expressing the requirement precisely; the agent must first produce something concrete enough to evaluate, which is refine. In the third case, reconstruct, the agent’s action changes how the user understands the problem and creates a judgment that did not exist when the interaction began, so the artifact itself helps the user think. All three make the initial prompt appear incomplete, yet they call for different behavior: reveal requires the agent to ask, refine requires it to act first and produce something evaluable, and reconstruct means that part of the delegation’s objective is created during the delegation itself.

Stronger models can reduce much of the cost of the first two processes. They can infer more latent preferences from context, store stable preferences in memory, and distinguish uncertainties that would change the next action from those that would not. Ambig-SWE measures this gap directly by converting SWE-Bench Verified into deliberately underspecified tasks that require an agent to notice missing critical information and obtain it through targeted clarification; models do not reliably know when to ask, and allowing interaction in this setting improved performance by as much as 74% relative to the non-interactive setting (Vijayvargiya et al., 2026). That gap can be trained, evaluated, and engineered. The third process cannot be eliminated through ex ante preference inference: if a judgment forms only after concrete consequences become visible, an earlier question cannot retrieve it. Cheap prototypes, reversible actions, and quick presentations of alternatives can lower the cost of forming the judgment, while the interaction itself still has to occur. A user profile created before execution is therefore insufficient because preferences continue to change before and after action, making personalization an iterative loop (Liang et al., 2026), and compressing different users’ utilities into one population-level reward can also produce an objective that represents none of them particularly well (Poddar et al., 2024).

Interactive benchmarks are beginning to reflect this shift. SWE-Together selected 109 repository-level tasks from 11,260 real user-agent coding sessions, using a reactive user simulator to preserve the original user’s intent and sequence of interventions while recording both final repository correctness and the number of corrective-feedback rounds required along the way (Wu et al., 2026). Once evaluation relies on a simulated user, however, the fidelity of that user model directly changes what the score means. A comparison involving 31 user simulators, 451 real participants, and 165 tasks found that LLM simulators were overly cooperative and stylistically narrow: they lacked the frustration and ambiguity of real users, gave consistently more positive feedback, and created an easy mode that inflated agent performance, while stronger models were not automatically more faithful simulators (Zhou et al., 2026). Once users enter the evaluation, the kind of user being modeled becomes an experimental parameter that requires calibration, a problem solver benchmarks never have to face.

Recognizing that goals form during execution changes how we view the rest of the delegation workflow. Even with solid constraint following, state tracking, and verification, a user who speaks only at the beginning will still underspecify the task. Because refine and reconstruct require a concrete artifact to inspect, forming the goal and reviewing the result are closely entangled in practice: when a person inspects a diff or a draft at the end of a run, they are often finishing the specification of what they wanted at the same time they are checking whether the agent built it properly. Measuring only final pass rates omits the human effort required to sustain that convergence, which leads directly to the cost of delegation itself.

Where the Cost of Delegation Goes

If AI cuts the time spent writing code in half, an engineer’s total time in real projects should fall substantially. The empirical data are less tidy, and the gap appears outside generation. On a well-bounded programming task with automatically executable tests, the treatment group using GitHub Copilot finished 55.8% faster on average than the control group (Peng et al., 2023). That experiment also clarifies the scope of the speedup: gains in generation speed are easiest to realize and measure when the task, context, and success criterion have already been fixed.

The result reversed in mature repositories familiar to the developers. In 2025, METR asked 16 experienced open-source developers to complete 246 real issues in their own repositories; with the AI tools available at the time, completion took roughly 19% longer on average. Before the study, participants expected to be 24% faster, and afterward they still believed they had been about 20% faster (METR, 2025). The distance among those three figures shows how easily participants’ sense of time spent can diverge from measured wall-clock time. METR’s later experiment produced a speedup in the raw estimates, although the confidence interval crossed zero and the participant and task mix had shifted substantially; METR therefore described the data as an unreliable signal of the current productivity effect and treated it only as weak evidence that tools may have become more helpful since 2025 (METR, 2026). Neither study establishes the overall direction of AI’s effect on developer productivity by itself; together they show that several costs outside generation remain easy to omit.

We can write those costs explicitly:

\[\text{Net Delegation Value} = U_{\text{verified outcome}} -\lambda_h C_{\text{human effort}} -\lambda_t C_{\text{latency}} -\lambda_c C_{\text{compute}} -\lambda_r E[\text{risk}]\]

Here $U_{\text{verified outcome}}$ is an outcome the user will accept, use, and treat as grounds for delegating again; $C_{\text{human effort}}$ includes prompting, context setup, monitoring, intervention, review, correction, and rework; and the remaining terms capture waiting, compute, and risk. Because an improvement in autonomy, pass rate, or latency moves only one term on the right-hand side, net value may not improve alongside it. For a set of delegations with similar task value and verification thresholds, we can examine the ratio between the first two terms directly by counting how many minutes all participants spend to obtain one outcome that genuinely counts, which I will call Human Minutes per Verified Valuable Outcome (HM/VVO):

\[\text{HM/VVO} = \frac{\sum_{i=1}^{N}\sum_{r\in R_i} m_{ir}}{\sum_{i=1}^{N} s_i}\]

Here $N$ is the number of delegations during the period, $R_i$ is the set of all roles involved in delegation $i$, $m_{ir}$ is the number of minutes role $r$ spends on understanding, clarification, oversight, verification, maintenance, and rework, and $s_i$ equals $1$ only when the outcome both passes verification and is actually adopted.

Three conditions must remain visible in this definition. First, the verification threshold for $s_i$ must scale with risk: a disposable script and an operation that changes production permissions require entirely different levels of confidence, and verification for the latter may cost more than generation itself. Before comparing systems, that threshold must be held constant; otherwise, reducing review makes the metric look better without producing an equally trustworthy outcome. Second, task values cannot vary too widely, since counting a low-value edit and a critical migration as equivalent outcomes deprives the ratio of meaning. Third, the numerator must include everyone’s time.

Decomposing $C_{\text{human effort}}$ one level further reveals three qualitatively different costs. One is spent steering the agent in the right direction: explaining the goal, supplying context, and asking or answering clarifying questions. Another is spent watching the process: seeing what the agent is doing, noticing drift, interrupting, correcting, and recovering. A third is spent after completion establishing confidence in the result: reading the diff, running checks, judging edge cases, and deciding whether the change is safe to adopt. Each cost requires a different remedy, and collapsing them into “agents are tiring to use” gives no clue about what to improve.

Early clarification primarily reduces misunderstanding, correction, and rework, while leaving the need for independent assurance unchanged. In a health-information setting, a randomized, blinded study rated Wayfinding AI, which actively requests context, as more helpful, relevant, and tailored than the baseline, and users were willing to wait longer in exchange for providing more context (Sayres et al., 2025). The study measured users’ evaluations of the answers and their willingness to wait rather than the direct exchange rate between one more early question and one less later revision; when an agent asks the wrong question, the question itself is pure overhead.

The shape of the second cost depends on whether the user can understand what the agent is doing. Interviews with 17 experienced developers divided oversight into a priori control, co-planning, real-time monitoring, and post-hoc review, documenting heuristics such as running tests or scanning a diff to reduce line-by-line reading (Dhanorkar et al., 2026). Those heuristics save attention only when their evidence actually supports continuing, rather than merely showing that the agent completed a local sub-goal it defined for itself. Without deliberate design, these costs surface together as coordination overhead: a study in which 31 participants used commercial agents on representative tasks recorded mental-model mismatch, failure to establish trust, incompatible collaboration styles, and excessive communication overhead (Shome et al., 2025), difficulties that fall into how users communicate goals, preferences, and constraints to agents and how agents communicate their capabilities, plans, actions, and side effects to users (Bansal et al., 2024). Without common ground, users cannot tell whether they have been understood or when they should intervene.

The third cost, post-completion review and assurance, is usually the hardest to compress. In $\tau$-bench, an agent converses with a simulated user in domains such as retail and airlines, calls domain-specific APIs, follows policy guidelines, and ends with a database state that is compared against an annotated goal state; alongside one-shot success, the benchmark measures repeated reliability with $\text{pass}^k$, and retail $\text{pass}^8$ is below 25% (Yao et al., 2025). A single success shows that the agent found one viable path, whereas a team needs sufficient assurance before it can delegate the same class of task reliably. Generation volume can grow with models and compute, while confidence is constrained by test coverage, CI, staged rollout, audit, rollback capacity, and the number of people who have time to review. Because generation capacity and trusted assurance capacity can grow at different rates, assurance can become the binding constraint on an organization’s ability to expand delegation (Figure 3).

Assurance keeps pace, or generation runs ahead Two panels show the same dependent task. Above, generation and assurance support every checkpoint through delivery and adoption. Each state becomes trusted and unlocks the next. Below, trusted progress stops at a subtle frontier while generation continues. Two speculative rollout paths continue downstream. One reaches human review and returns toward the frontier as rework while already generated downstream work becomes invalid. Another waits, undergoes review and rework, and may eventually be abandoned. The diagram is schematic and contains no measured quantities. assurance keeps pace task generation assurance delivery adoption generation outruns assurance task generation assurance trusted frontier human review rework waiting ↻ rework abandon
Figure 3. Assurance determines which generated state can safely support the next dependent step. Above, generation and evidence advance together through delivery and adoption. Below, generation continues beyond the trusted frontier, producing speculative rollouts, review and rework while trusted progress remains fixed; downstream work may already exist when an upstream judgment invalidates it. Schematic, with no measured quantities.

If a person has to inspect every step, tool call, and diff with equal intensity, more autonomous agents create more work for people. The engineering approach to reducing this cost is the runtime expression of supervision leverage: OpenAI’s account of building an agent-first software product treats human time and attention as scarce resources, placing more domain context, architectural constraints, observability, and verification mechanisms into the repository and harness so that agents can operate over a larger problem space (Lopopolo, 2026). People spend less effort producing and checking code line by line, and more effort constructing an environment in which code can be generated, checked, and recovered reliably, work that also disappears from the cost accounting of any single delegation.

As compilers, linters, and CI suites filter out obvious breakage automatically, what reaches a human reviewer changes shape. The outputs that land on a person’s desk are typically the ones that look right on the surface: the code compiles, the tests pass, and the agent reports completion. Reviewing work that already passes automated checks remains demanding because the mechanical errors have already been stripped away, leaving three judgments that automated tests cannot settle:

  1. Whether the result matches what was actually wanted. Automated checks verify compliance with a written test or specification. If the agent wrote the tests itself under a flawed understanding of the requirement, implementation and test may drift together. Even with an independent test suite, inspecting the concrete patch is often the moment when the user realizes that an unstated preference was missed or that a technically valid solution solved the wrong problem.
  2. Whether the implementation is worth keeping in the system. A patch can pass functional checks while introducing brittle structure or omitting operational basics that make future maintenance harder. A study of 4,550 agentic pull requests across 81 open-source repositories found that agents handle logging differently from human developers, rarely adding sufficient logging on their own, and that humans ultimately performed 72.5% of post-generation log repairs, acting as what the authors call “silent janitors” who clean up observability problems left by agents even when review feedback never mentioned the requirement (Ouatiti et al., 2026). The study measures who repairs logs; reading it as “the initiator receives the benefit while downstream maintainers absorb the cost” is my inference, though it matches the experience of many teams. That is why HM/VVO sums over $r$: at least three roles may sit inside $R_i$, namely the requester who initiates the delegation, the reviewer who checks it, and the downstream maintainer who inherits it. Counting only the requester’s time and stopping at delivery is the accounting choice that hides the downstream cost.
  3. Whether the humans responsible for the system still understand how it works. In any codebase or workflow that outlives a single prompt, someone has to debug failures, extend functionality, and take responsibility when incidents occur. When reviewers treat a green test run as sufficient to merge large changes they have not digested, the team’s grasp of its own system erodes. Reviewing a substantive change serves both to catch defects and to help the human owners absorb what changed so they remain able to maintain the system afterward.

Which Loop Should the Human Be On?

If step-by-step babysitting wastes attention and waiting until the very end risks reviewing a large, test-passing artifact built in the wrong direction, where should human judgment actually sit? Many products still implement “human in the loop” either as an approval prompt before every tool call or as a single sign-off at the end. Both designs treat human attention as uniform. In practice, people enter an agent’s trajectory for four distinct reasons (Figure 4): low-risk, reversible actions backed by strong automated checks can proceed directly; choices that hinge on unstated user preferences or private context warrant a clarifying interaction; actions that touch money, permissions, production state, security, or other irreversible boundaries require explicit authorization; and runs where verifiers conflict, state looks corrupted, or the agent cannot diagnose a failure call for escalation and human takeover.

Where human attention enters an agent trajectory Two panels share the same grid of ten action opportunities and the same two lanes, a human lane above and an agent lane below, named in the left margin. A single heavy line, the control path, shows who holds control at each moment, and a strip underneath each panel records that control state; a legend at the top right gives the strip's three readings, agent, human and waiting. In the upper panel, naive human-in-the-loop, the control path stays on the agent lane but is broken at every one of the ten actions by a gate, and a dashed request rises from each gate to the human lane; the strip below reads agent throughout but is cut open at all ten gates, so control never moves although the trajectory stops constantly. In the lower panel, selective human involvement, the control path runs along the agent lane and is joined at the second action by a dashed round trip up to the human lane and back, labelled clarification and information. At the third action the path launches two lighter curved branches, labelled sub-agents, which run below the agent lane and rejoin at the fourth and fifth actions; control does not move for either. The path is then stopped at one gate at the sixth action, labelled approval and authority. At the seventh action the control path itself rises to the human lane and turns neutral, labelled escalation and takeover, and comes back down at the ninth action, labelled handback with goal, state and authority. The strip below shows three spans, agent then human then agent, with a single cut at the approval gate. agent human waiting naive human-in-the-loop human agent control selective human involvement human agent control sub-agents clarification information approval authority escalation takeover handback goal, state, authority
Figure 4. Where human attention enters an agent trajectory. The heavy line is the control path, and its height shows who holds control: accented while the agent drives, neutral while the human drives. A gate is an action that cannot run until someone with the necessary authority allows it. The control strip is cut there because a trajectory waiting for approval is in neither state named by the strip. A dashed round trip to the human lane requests information without moving control. The same is true of sub-agents launched under the agent lane, which show what direct execution actually looks like. Control changes hands only when the path itself leaves the agent lane, and it can return with a revised goal, corrected state, and the authority required for the next stretch. The upper panel stops at every action and never transfers control.

Interrupting the user whenever the model is uncertain is a poor default, because every question taxes human attention and most answers do not change what the agent should do next. A more disciplined criterion weighs the value of information against the cognitive cost of interrupting the user, taking into account task risk, ambiguity, and the effort required to answer (Dong et al., 2026). That trade-off explains why a chatty agent can feel just as exhausting as a reckless one. Deciding when to ask is a distinct skill that does not automatically improve alongside raw problem solving: on HiL-Bench, which hides missing, ambiguous, or contradictory requirements inside software engineering and text-to-SQL tasks so they surface only during exploration, models that perform well with complete specifications recover only part of their score when they must decide on their own when to call ask_human() (Elfeki et al., 2026).

Verbal questions are also only one way to get human input, and often an inefficient one when the user needs to see something concrete to know what they want. Putting a cheap prototype, a reversible edit, a draft pull request, or a dry-run preview in front of the user early in the trajectory can resolve refine and reconstruct far faster than an abstract questionnaire. Showing an early, low-cost artifact also directly reduces the post-completion review burden described in the previous section: catching a framing mistake on a fifty-line sketch spares the reviewer from untangling a two-thousand-line implementation that already passes all its tests. Because this tactic relies on being able to throw the draft away cleanly, reversibility acts both as a safety boundary and as a design lever that lets concrete previews replace rounds of questioning.

Where irreversible actions are concerned, capability gains do not erase the boundary between prediction and permission. A model may grow arbitrarily good at predicting which actions a user would approve, yet predicting an approval and holding the authority to act remain different things. At the same time, requiring a human to authorize an action does not require making them click a button on every step. Defining explicit scopes of authority in advance, specifying which classes of actions may run automatically, under what preconditions, and within what blast radius, lets routine work proceed uninterrupted while reserving live approval for actions that cross the boundary.

Experienced agent users already allocate their attention in this selective way. Telemetry across millions of human-agent interactions in Claude Code and Anthropic’s public API shows that more experienced users enable auto-approval more often while also interrupting the agent more frequently (Anthropic, 2026). For these users, autonomy and oversight do not sit at opposite ends of a single slider: seasoned users stop clicking routine permission prompts and intervene more actively when something matters. Although the telemetry does not record the exact reason for each interruption, the pattern is consistent with users shifting their effort away from mechanical step-by-step confirmation toward high-leverage course corrections. Systems such as Magentic-UI build these distinct entry points directly into the interface through upfront co-planning, live co-tasking and browser takeover, action guards, and answer verification (Mozannar et al., 2025).

Concentrating human involvement into fewer, higher-stakes moments carries a structural trap of its own. Across 106 studies and 370 effect sizes, a meta-analysis of human-AI collaboration found that combined human-AI teams performed worse on average than the stronger of the human or the AI working alone, particularly on decision-making tasks, with positive gains depending heavily on how tasks and information were divided (Vaccaro et al., 2024). Simply inserting a person into the workflow guarantees nothing. Worse, as Lisanne Bainbridge observed in her classic study of industrial and aviation automation, turning human operators into passive monitors of a mostly reliable system erodes the very hands-on familiarity and skill they need when forced to step in during rare, complex failures (Bainbridge, 1983). This connects directly back to why human reviewers have to digest the changes an agent makes: if automation handles routine runs and humans only glance at green checkmarks, the people nominally in charge steadily lose their mental model of the system, leaving them ill-equipped to judge the hardest edge cases or recover when an autonomous run goes off the rails.

The Human Is Part of the Agent

Step back from these individual stages, and a consistent pattern emerges: the conditions that determine whether a delegation succeeds mostly sit outside the software box we call “the agent.” A benchmark score omits the human judgment spent crafting the task and its verifier. A product metric that logs “task completed” omits the time a reviewer spends reading the diff and the time a maintainer spends cleaning up logs months later. The agent runtime itself does not decide who holds organizational authority or who answers for a production incident. From the perspective of the person typing the prompt, the teammates who absorb the downstream review and maintenance work may be out of sight as well. In every case, the system that actually produces a working, trusted result is larger than the agent alone.

Rapid gains in model capability can create the illusion that human involvement is simply vanishing, when much of it is changing position. The evolution of training and evaluation already showed how sample-level grading gave way to writing principles, building verifiers, and designing environments. A parallel shift is now playing out at runtime. As models and harnesses get better at execution control, including remembering constraints across long contexts, checking workspace state, gathering evidence before editing, running tests, and rolling back failed attempts, much of today’s babysitting will fade away. It should fade away; compensating for brittle execution control is unfinished engineering rather than a lasting human role.

Once that operational babysitting recedes, human work separates into two distinct locations: what can be specified in advance moves upstream before execution begins, while the judgments that depend on inspecting a concrete artifact remain during the run and at delivery.

Moving upstream before runtime are the repeatable rules of the game: architectural constraints, coding standards, automated test suites, sandbox boundaries, predefined scopes of authority, and clear lines of organizational accountability across operators, engineering owners, managers, vendors, and regulators. Putting those guardrails in place ahead of time removes the need for repetitive per-action clicks and ensures that when an incident occurs, liability does not fall by default on whoever happened to press an Approve button last.

Remaining during the trajectory and at final review are the tasks that no upfront rule or automated test suite can finish on its own:

  • Forming the rest of the goal through concrete artifacts. Because people often discover what they want and what trade-offs they accept only after seeing a working draft, agents need to surface low-cost, reversible previews at key decision forks so direction can be corrected before thousands of lines of polished work accumulate.
  • Reviewing work that already looks right on the surface. As automated checks catch syntax errors, broken builds, and obvious regressions before a human ever looks, human review concentrates on what passes the test suite: verifying that the solution addresses the real underlying intent, judging whether the code or artifact is clean and maintainable enough for downstream teammates to inherit, and understanding the change well enough to own and debug it in production.
  • Reconstructing context when an autonomous run escalates. The longer an agent runs without routine human involvement, the less situational awareness the human retains when an anomaly occurs or a complex deliverable lands for sign-off. As routine interruptions decline, the central bottleneck shifts from whether people are willing to step in to whether they can understand the situation in time. That makes the quality of the handoff a core design problem for agent products: presenting the relevant history, current state, what automated checks have already verified, the open trade-offs, downstream impact, reversibility, and required authority in a form a human can actually evaluate, where most tools today still offer little more than a bare Allow or Deny prompt.

Evaluations are gradually expanding to reflect this wider boundary. $\tau$-bench asks whether a workflow is reliable across repeated runs; HiL-Bench tests whether an agent knows when to pause and ask; SWE-Together counts how many rounds of human course correction a trajectory takes. Each of these benchmarks runs into the same measurement challenge that teams face in production: once the human is recognized as part of the system, what we are evaluating and designing is the joint machinery through which human judgment and model execution turn an initial intent into a result someone can understand, trust, and stand behind.

References 38 sources

  1. The 2026 AI Index Report: Technical PerformanceStanford Institute for Human-Centered Artificial Intelligence · Stanford University · 2026
  2. Our First Proof submissionsOpenAI · 2026
  3. Ten advances in mathematics and theoretical computer scienceOpenAI · 2026
  4. Assessing Claude Mythos Preview’s cybersecurity capabilitiesAnthropic · 2026
  5. Project Glasswing: Securing critical software for the AI eraAnthropic · 2026
  6. How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World SessionsTang et al. · arXiv · 2026
  7. Coding Agents Don’t Know When to ActGloaguen et al. · arXiv · 2026
  8. SWE-Touch: Benchmarking Coding Agents When Users Touch the CodeTan et al. · arXiv · 2026
  9. AgentRx: Diagnosing AI Agent Failures from Execution TrajectoriesBarke et al. · arXiv · 2026
  10. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?Jimenez et al. · arXiv · 2023
  11. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?Miserendino et al. · arXiv · 2025
  12. DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering TasksHuang et al. · arXiv · 2026
  13. Training Language Models to Follow Instructions with Human FeedbackOuyang et al. · NeurIPS · 2022
  14. ChatGPT Outperforms Crowd-Workers for Text-Annotation TasksGilardi, Alizadeh, and Kubli · arXiv · 2023
  15. Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaZheng et al. · arXiv · 2023
  16. Constitutional AI: Harmlessness from AI FeedbackBai et al. · arXiv · 2022
  17. Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE SessionsTang et al. · arXiv · 2026
  18. Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software EngineeringVijayvargiya et al. · ICLR · 2026
  19. Learning Personalized Agents from Human FeedbackLiang et al. · arXiv · 2026
  20. Personalizing Reinforcement Learning from Human Feedback with Variational Preference LearningPoddar et al. · arXiv · 2024
  21. SWE-Together: Evaluating Coding Agents in Interactive User SessionsWu et al. · arXiv · 2026
  22. Mind the Sim2Real Gap in User Simulation for Agentic TasksZhou et al. · arXiv · 2026
  23. The Impact of AI on Developer Productivity: Evidence from GitHub CopilotPeng et al. · arXiv · 2023
  24. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer ProductivityMETR · arXiv · 2025
  25. We are Changing our Developer Productivity Experiment DesignMETR · 2026
  26. Towards Better Health Conversations: The Benefits of Context-seekingSayres et al. · arXiv · 2025
  27. Human Oversight of Agentic Systems in Practice: Examining the Oversight Work, Challenges, and Heuristics of Developers Using Software AgentsDhanorkar, Passi, and Vorvoreanu · arXiv · 2026
  28. Why Johnny Can’t Use Agents: Industry Aspirations vs. User Realities with AI Agent SoftwareShome, Krishnan, and Das · arXiv · 2025
  29. Challenges in Human-Agent CommunicationBansal et al. · arXiv · 2024
  30. \τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsYao et al. · ICLR · 2025
  31. Harness Engineering: Leveraging Codex in an Agent-First WorldLopopolo · openai.com · 2026
  32. Do AI Coding Agents Log Like Humans? An Empirical StudyOuatiti et al. · arXiv · 2026
  33. Value of Information: A Framework for Human-Agent CommunicationDong et al. · arXiv · 2026
  34. HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?Elfeki et al. · arXiv · 2026
  35. Measuring AI Agent Autonomy in PracticeAnthropic · 2026
  36. Magentic-UI: Towards Human-in-the-Loop Agentic SystemsMozannar et al. · arXiv · 2025
  37. When Combinations of Humans and AI Are Useful: A Systematic Review and Meta-AnalysisVaccaro, Almaatouq, and Malone · Nature Human Behaviour · 2024
  38. Ironies of AutomationBainbridge · Automatica · 1983

Back to the blog