Over the past few years, the field has tracked model capability by building ever harder benchmarks. When MMLU stopped separating frontier models, researchers introduced GPQA; as competition mathematics approached saturation, evaluations moved to harder AIME and IMO problems and then to Humanity’s Last Exam; coding benchmarks progressed from HumanEval to SWE-bench, Terminal-Bench, and software engineering tasks that unfold over much longer time horizons. Keeping pace has grown difficult. The Stanford 2026 AI Index notes that some benchmarks intended to last for years began approaching saturation within months, and on Humanity’s Last Exam, which was designed specifically for frontier models and weighted toward questions that challenge human experts, the top score rose by roughly 30 percentage points in a single year (Stanford Institute for Human-Centered Artificial Intelligence, 2026). Extending the ruler makes sense for what these suites measure: a set of repeatable, objectively scored problems offers a tractable proxy for how capable a model has become.
As those suites grow harder, model scores are reaching familiar human reference points such as olympiad medals, domain expertise, and open research problems. In July 2025, OpenAI’s general-purpose reasoning model scored 35 out of 42 on the International Mathematical Olympiad, reaching gold-medal level. By February 2026, experts judged at least five of ten proof attempts in First Proof likely to be correct, including problems that had remained open for years before their authors found solutions (OpenAI, 2026). In August 2026, ten announced results in mathematics and theoretical computer science each solved or substantially advanced a longstanding open problem (OpenAI, 2026). Those ten results involved multiple attempts, human selection, and expert feedback, so treating every one as an independent model achievement would be inaccurate, even as the direction of capability growth remains clear. In cybersecurity, the same capability curve carries immediate operational consequences. Mythos Preview can discover and exploit zero-day vulnerabilities in major operating systems and browsers, finding many previously unknown vulnerabilities autonomously after an initial prompt (Anthropic, 2026). Because that capability is asymmetric between offense and defense, programs such as Project Glasswing give defenders earlier access (Anthropic, 2026).
Following this curve alone suggests that agents have become extraordinarily smart. They solve mathematical problems beyond the reach of most people, contribute at research frontiers accessible to only a small number of domain experts, and outperform human experts on some highly specialized tasks. On many frontier tasks, an ordinary person has moved from asking whether they would do better than the model to wondering whether they can even follow what it is doing. Yet daily work with Claude Code, Codex, or any comparably capable agent produces a more complicated impression. The model is unquestionably powerful, sometimes absurdly so, and yet on tasks that appear far easier than olympiad mathematics or vulnerability research it can still occasionally feel strangely stupid. A task may finish, the code may run, and the output may contain no identifiable factual error, while something still feels wrong. Switching models, changing the interaction style, or adding a few turns of back-and-forth can change how pleasant the agent is to use in ways that benchmark accuracy does not capture.
That gap arises because the single word “smart” is being asked to evaluate five different system properties. A benchmark primarily measures problem-solving capability: given a fully specified problem, how difficult a problem can the solver handle? Everyday delegation exposes four additional questions. Agentic control asks whether the agent can apply its capability reliably across a long trajectory. Task formation asks whether the system can continue forming and revising the task through interaction. Assurance asks whether there is sufficiently independent evidence to justify adopting the result. Governance asks who authorizes an action and how responsibility is assigned when something goes wrong. All five can overlap in a single failure, and they improve at different rates (Figure 1). When all five are compressed into one judgment about whether the agent is smart, benchmark scores can keep rising while day-to-day usability fails to improve at the same pace.
What, Exactly, Feels Stupid?
When people complain about an agent in daily work, they rarely mean that the model lacks reasoning ability. Typical complaints sound much more specific: “You knew that constraint, so why did you violate it again?”, “You started changing things without confirming anything,” “I just changed that, didn’t you notice?”, “You said it was done without verifying it,” or “You made a mess and can’t roll it back.” Every complaint assumes that the model was capable of doing the right thing; the failure lies in how it managed that capability across an unfolding trajectory. Grouping these complaints by the system function they implicate shows that they arise at different layers and require different remedies:
| What the user feels | A more precise description | Primary system function |
|---|---|---|
| “You knew that constraint. Why did you violate it again?” | constraint persistence | agentic control |
| “You started changing things without confirming anything.” | evidence threshold | agentic control |
| “There was nothing to change, yet you changed it anyway.” | action bias | agentic control |
| “You forgot your own plan halfway through.” | plan adherence | agentic control |
| “You said it was done without verifying it.” | self-report calibration | agentic control |
| “Your changes keep expanding and you won’t stop.” | scope control | agentic control |
| “I just changed that. Didn’t you notice?” | shared state reconciliation | agentic control |
| “You made a mess and can’t roll it back.” | recoverability | agentic control |
| “You wrote the tests and then used them to validate your own work.” | non-independent verification | assurance |
| “You would rather guess than ask me.” | help-seeking calibration | task formation |
| “I had to see the result before I knew I didn’t want it this way.” | objective reconstruction | task formation |
| “You had no authority to make that decision for me.” | authority | governance |
The first eight rows mostly concern agentic control, which model training and the harness must address together. A constraint or plan may drop out of the context after dozens of steps, or it may remain present without being followed; most failures involve some combination of both, so assigning every row exclusively to one side is rarely clean. The ninth row crosses into assurance. Treating “you wrote the tests and then used them to validate your own work” as an assurance problem is my own judgment rather than a claim directly established in the literature: if the agent has misunderstood the requirement, its tests may drift in the same direction as its implementation, so a green test run provides no independent evidence. The tenth and eleventh rows belong to task formation, covering information that has not yet been obtained and a judgment that has not yet been formed. The final row belongs to governance, which determines which actions require approval and from whom. Although real failures often span multiple rows, the table identifies the first problem each complaint reveals.
Most of the empirical evidence below comes from coding agents. Software engineering provides executable tests, git, diffs, CI, sandboxes, and rollback, which make failures unusually visible, localizable, and countable compared with legal, medical, financial, or organizational decisions, where correctness may itself be ambiguous and reversal difficult. The coding literature is useful for identifying mechanisms, though its measured proportions should not be generalized directly to every industry.
Observational logs show how common these failures are in practice. An analysis of 20,574 coding-agent sessions across 1,639 repositories identified seven recurring forms of misalignment spanning project understanding, intent interpretation, rule adherence, action boundaries, code execution, and progress reporting. In 90.5% of episodes, the failure imposed a cost on effort or trust. Because most sessions did not confirm the final outcome in the visible conversation, resolution could be observed in only 9.33% of episodes (1,504 cases); among those, 91.49% required explicit user correction (Tang et al., 2026). As the overall problem rate declined in the dataset, constraint violations and inaccurate self-reports made up a larger share of what remained. Observational data cannot establish that stronger models caused this shift, though it suggests that the remaining problems increasingly fall outside a simple inability to perform the task.
Controlled experiments reproduce two of these patterns directly. FixedBench tests whether an agent knows when to stop: all 200 of its tasks come from issues that had already been fixed, so the correct action is to recognize that fact and leave the code unchanged. Across five recent models and four harnesses, agents still proposed unnecessary changes in 35% to 65% of cases. Requiring them to reproduce the problem before patching reduced some over-action while creating under-action on issues that had been only partially fixed (Gloaguen et al., 2026). FixedBench does not establish why agents are so inclined to act; one plausible explanation is that in most training and evaluation signals they have seen, doing nothing is rarely counted as success. A second benchmark, SWE-Touch, tests what happens when the environment changes while the agent is working. In real development, users read, edit, and sometimes revert code during an agent run, so the repository state is jointly produced. Injecting simulated user edits relevant to the task and capable of changing the completion path into SWE-bench Verified reduced the average resolve rate by 7.7 percentage points, with trajectory analysis showing agents retaining code that conflicted with user edits or overwriting those edits without rechecking the workspace or running targeted tests (Tan et al., 2026). Standard SWE-bench never exposes this behavior because nobody else modifies its environment.
Final pass or fail labels also reveal little about where an agent first went off course. Human annotation of 115 failed trajectories across structured API workflows, incident management, and open-ended web and file tasks identified nine failure categories, including plan-adherence failure, invalid invocation, misreading tool output, intent-plan misalignment, and underspecified user intent, while attempting to locate the first unrecoverable critical step in each trajectory (Barke et al., 2026). The same failure modes recur across long trajectories even when the task has nothing to do with writing code.
Calling these breakdowns “the model cannot do the task” has become increasingly misleading. The model is usually capable of completing each local step; the breakdown happens over dozens or hundreds of steps, when it must still honor original constraints, maintain current state, gather evidence before acting when evidence is weak, control scope, know when to stop, verify the outcome, report status honestly, and recover after a mistake. This layer of agentic control is distinct from the difficulty of the problems a model can solve, and it is the target of many complaints that begin, “You clearly know how, so why did you do that?” I expect this layer to improve substantially because each component offers concrete engineering levers: persistent constraints, refreshed workspace state, pre-action evidence thresholds, automated verification, checkpoints, and rollback can all be addressed through the harness, context management, and training incentives. That remains a prediction without long-term evidence, and FixedBench’s under-action trade-off shows that reducing one control failure can produce another. A more defensible expectation is that a meaningful share of today’s perceived stupidity will decline as systems improve. That leaves a natural question: if control failures are so common in real use, why are they nearly invisible on benchmarks?
What Benchmarks Fix for the Agent
Benchmarks largely miss these problems by design, and that design is appropriate for what they measure. SWE-bench gives an agent a real GitHub repository, an issue description, and a code environment, then uses tests to determine whether the resulting patch works (Jimenez et al., 2023). SWE-Lancer extends the same idea to more than 1,400 tasks drawn from real freelance work and maps them to roughly one million dollars in actual payouts (Miserendino et al., 2025). Both answer a solver question: given a written task and a fixed success criterion, how difficult a problem can the model solve? Meaningful measurement requires the task, environment, and verifier to remain fixed. In real delegation, a larger system must continually supply and maintain those conditions across the agent, harness, user, CI pipeline, and organizational process. The task may be reinterpreted, other people may modify the environment concurrently, success may depend on preferences the user has not articulated, and validation may come from an independent checker or reviewer. Because a benchmark fixes these variables on behalf of the whole system for the duration of a run, a component capability score cannot by itself predict how the same model will perform in a real delegation.
Preparing those clean benchmark conditions takes substantial human work beforehand, and it is expensive. DeepSWE designed 113 long-horizon software engineering tasks from scratch across 91 real repositories and hand-wrote a behavior-oriented verifier for each task, accepting implementations the researchers had never seen so long as they satisfied the requested functionality. Independent LLM judges disagreed with those verifiers only 1.4% of the time, compared with 32.4% disagreement on tests inherited from SWE-Bench Pro (Huang et al., 2026). The scarce human judgment in DeepSWE sits upstream of solving the 113 tasks: deciding which tasks are worth measuring, which complexities of a real environment should remain, and what an unfamiliar solution must satisfy to count as success.
This shift upstream began well before modern agent evaluation. In early RLHF, labelers wrote ideal responses to user prompts for supervised fine-tuning, ranked multiple model-generated candidates, trained a reward model on those rankings, and used it to optimize the policy (Ouyang et al., 2022), spending most human effort on individual samples. Models later absorbed much of that sample-level work. In 2023, ChatGPT outperformed crowdworkers on four of five text-annotation task categories, achieved higher inter-coder agreement, and cost roughly one twentieth as much per item as MTurk (Gilardi et al., 2023), while GPT-4 acting as a judge agreed with human preferences more than 80% of the time, with systematic costs such as position and verbosity bias (Zheng et al., 2023). Because the work being absorbed consisted of bounded judgments attached to single samples, human effort could move toward judgments with broader coverage. Constitutional AI made this shift explicit: humans write a set of principles, and the model uses the constitution to critique a response, identify violations, revise it, and compare responses to generate preference data for reinforcement learning (Bai & others, 2022). Instead of judging every response individually, a human can write one principle that constrains an entire class of future cases.
Across these different research threads, a common dimension emerges: how much future behavior a single human judgment can constrain, which I will call supervision leverage (Figure 2):
Toward the left of the spectrum, the human produces or grades individual answers for the model; toward the right, the human defines the problem space in which the model operates and what success means inside it. A label affects one sample, a principle constrains a class of outputs, a verifier defines a set of acceptable solutions, and the task and environment establish the boundary of the entire problem space. This spectrum describes judgment coverage; by itself, it says nothing about whether total human hours rise or fall. It supports a narrower conclusion: some sample-level labor is declining while judgment moves into principles, verifiers, task design, and environment design. Those new positions often fall outside the scoring boundary: a SWE-bench score does not account for the judgment spent designing its environment, just as a successful delegation does not account for all the context a user supplied to make it work.
This progression still rests on an underlying assumption: that the task, rubric, and objective can be fully specified before the agent begins. Benchmarks turn that assumption into a design condition, which is how they measure solver capability reliably. User-facing agents rarely receive such clean conditions.
We May Have Been Evaluating a User Who Does Not Exist
Standard evaluations contain an unusually cooperative user: someone who states the full requirement at once, provides all relevant context, defines the success criterion upfront, and holds the same judgment throughout the task. They speak once at the beginning, and tests, a verifier, or a judge determine correctness afterward. Isolating problem-solving capability requires this kind of user, yet real work rarely provides one.
Consider an engineer who says, “This feature has become too complicated. Simplify it.” At the start, they may be unable to state how simple is simple enough. After seeing the first revision, they realize that their actual concern is whether a new teammate can understand the code, and that some apparently redundant branches encode lessons from earlier production incidents. The original request was valid, yet it needed a concrete alternative before it could become specific enough. A behavioral analysis of 11,579 AI-assisted IDE sessions found the same pattern: developers rarely state everything at once, tightening the objective instead through a sequence of short prompts and changing direction after seeing intermediate results (Tang et al., 2026).
Three different processes sit inside that back-and-forth. Sometimes the user knows what matters from the start and the agent simply lacks that information; the agent needs to ask, which we can call reveal. Sometimes the user has only a rough direction and needs to see an implementation and its trade-offs before expressing the requirement precisely; the agent must first produce something concrete enough to evaluate, which is refine. In the third case, reconstruct, the agent’s action changes how the user understands the problem and creates a judgment that did not exist when the interaction began, so the artifact itself helps the user think. All three make the initial prompt appear incomplete, yet they call for different behavior: reveal requires the agent to ask, refine requires it to act first and produce something evaluable, and reconstruct means that part of the delegation’s objective is created during the delegation itself.
Stronger models can reduce much of the cost of the first two processes. They can infer more latent preferences from context, store stable preferences in memory, and distinguish uncertainties that would change the next action from those that would not. Ambig-SWE measures this gap directly by converting SWE-Bench Verified into deliberately underspecified tasks that require an agent to notice missing critical information and obtain it through targeted clarification; models do not reliably know when to ask, and allowing interaction in this setting improved performance by as much as 74% relative to the non-interactive setting (Vijayvargiya et al., 2026). That gap can be trained, evaluated, and engineered. The third process cannot be eliminated through ex ante preference inference: if a judgment forms only after concrete consequences become visible, an earlier question cannot retrieve it. Cheap prototypes, reversible actions, and quick presentations of alternatives can lower the cost of forming the judgment, while the interaction itself still has to occur. A user profile created before execution is therefore insufficient because preferences continue to change before and after action, making personalization an iterative loop (Liang et al., 2026), and compressing different users’ utilities into one population-level reward can also produce an objective that represents none of them particularly well (Poddar et al., 2024).
Interactive benchmarks are beginning to reflect this shift. SWE-Together selected 109 repository-level tasks from 11,260 real user-agent coding sessions, using a reactive user simulator to preserve the original user’s intent and sequence of interventions while recording both final repository correctness and the number of corrective-feedback rounds required along the way (Wu et al., 2026). Once evaluation relies on a simulated user, however, the fidelity of that user model directly changes what the score means. A comparison involving 31 user simulators, 451 real participants, and 165 tasks found that LLM simulators were overly cooperative and stylistically narrow: they lacked the frustration and ambiguity of real users, gave consistently more positive feedback, and created an easy mode that inflated agent performance, while stronger models were not automatically more faithful simulators (Zhou et al., 2026). Once users enter the evaluation, the kind of user being modeled becomes an experimental parameter that requires calibration, a problem solver benchmarks never have to face.
Recognizing that goals form during execution changes how we view the rest of the delegation workflow. Even with solid constraint following, state tracking, and verification, a user who speaks only at the beginning will still underspecify the task. Because refine and reconstruct require a concrete artifact to inspect, forming the goal and reviewing the result are closely entangled in practice: when a person inspects a diff or a draft at the end of a run, they are often finishing the specification of what they wanted at the same time they are checking whether the agent built it properly. Measuring only final pass rates omits the human effort required to sustain that convergence, which leads directly to the cost of delegation itself.
Where the Cost of Delegation Goes
If AI cuts the time spent writing code in half, an engineer’s total time in real projects should fall substantially. The empirical data are less tidy, and the gap appears outside generation. On a well-bounded programming task with automatically executable tests, the treatment group using GitHub Copilot finished 55.8% faster on average than the control group (Peng et al., 2023). That experiment also clarifies the scope of the speedup: gains in generation speed are easiest to realize and measure when the task, context, and success criterion have already been fixed.
The result reversed in mature repositories familiar to the developers. In 2025, METR asked 16 experienced open-source developers to complete 246 real issues in their own repositories; with the AI tools available at the time, completion took roughly 19% longer on average. Before the study, participants expected to be 24% faster, and afterward they still believed they had been about 20% faster (METR, 2025). The distance among those three figures shows how easily participants’ sense of time spent can diverge from measured wall-clock time. METR’s later experiment produced a speedup in the raw estimates, although the confidence interval crossed zero and the participant and task mix had shifted substantially; METR therefore described the data as an unreliable signal of the current productivity effect and treated it only as weak evidence that tools may have become more helpful since 2025 (METR, 2026). Neither study establishes the overall direction of AI’s effect on developer productivity by itself; together they show that several costs outside generation remain easy to omit.
We can write those costs explicitly:
\[\text{Net Delegation Value} = U_{\text{verified outcome}} -\lambda_h C_{\text{human effort}} -\lambda_t C_{\text{latency}} -\lambda_c C_{\text{compute}} -\lambda_r E[\text{risk}]\]Here $U_{\text{verified outcome}}$ is an outcome the user will accept, use, and treat as grounds for delegating again; $C_{\text{human effort}}$ includes prompting, context setup, monitoring, intervention, review, correction, and rework; and the remaining terms capture waiting, compute, and risk. Because an improvement in autonomy, pass rate, or latency moves only one term on the right-hand side, net value may not improve alongside it. For a set of delegations with similar task value and verification thresholds, we can examine the ratio between the first two terms directly by counting how many minutes all participants spend to obtain one outcome that genuinely counts, which I will call Human Minutes per Verified Valuable Outcome (HM/VVO):
\[\text{HM/VVO} = \frac{\sum_{i=1}^{N}\sum_{r\in R_i} m_{ir}}{\sum_{i=1}^{N} s_i}\]Here $N$ is the number of delegations during the period, $R_i$ is the set of all roles involved in delegation $i$, $m_{ir}$ is the number of minutes role $r$ spends on understanding, clarification, oversight, verification, maintenance, and rework, and $s_i$ equals $1$ only when the outcome both passes verification and is actually adopted.
Three conditions must remain visible in this definition. First, the verification threshold for $s_i$ must scale with risk: a disposable script and an operation that changes production permissions require entirely different levels of confidence, and verification for the latter may cost more than generation itself. Before comparing systems, that threshold must be held constant; otherwise, reducing review makes the metric look better without producing an equally trustworthy outcome. Second, task values cannot vary too widely, since counting a low-value edit and a critical migration as equivalent outcomes deprives the ratio of meaning. Third, the numerator must include everyone’s time.
Decomposing $C_{\text{human effort}}$ one level further reveals three qualitatively different costs. One is spent steering the agent in the right direction: explaining the goal, supplying context, and asking or answering clarifying questions. Another is spent watching the process: seeing what the agent is doing, noticing drift, interrupting, correcting, and recovering. A third is spent after completion establishing confidence in the result: reading the diff, running checks, judging edge cases, and deciding whether the change is safe to adopt. Each cost requires a different remedy, and collapsing them into “agents are tiring to use” gives no clue about what to improve.
Early clarification primarily reduces misunderstanding, correction, and rework, while leaving the need for independent assurance unchanged. In a health-information setting, a randomized, blinded study rated Wayfinding AI, which actively requests context, as more helpful, relevant, and tailored than the baseline, and users were willing to wait longer in exchange for providing more context (Sayres et al., 2025). The study measured users’ evaluations of the answers and their willingness to wait rather than the direct exchange rate between one more early question and one less later revision; when an agent asks the wrong question, the question itself is pure overhead.
The shape of the second cost depends on whether the user can understand what the agent is doing. Interviews with 17 experienced developers divided oversight into a priori control, co-planning, real-time monitoring, and post-hoc review, documenting heuristics such as running tests or scanning a diff to reduce line-by-line reading (Dhanorkar et al., 2026). Those heuristics save attention only when their evidence actually supports continuing, rather than merely showing that the agent completed a local sub-goal it defined for itself. Without deliberate design, these costs surface together as coordination overhead: a study in which 31 participants used commercial agents on representative tasks recorded mental-model mismatch, failure to establish trust, incompatible collaboration styles, and excessive communication overhead (Shome et al., 2025), difficulties that fall into how users communicate goals, preferences, and constraints to agents and how agents communicate their capabilities, plans, actions, and side effects to users (Bansal et al., 2024). Without common ground, users cannot tell whether they have been understood or when they should intervene.
The third cost, post-completion review and assurance, is usually the hardest to compress. In $\tau$-bench, an agent converses with a simulated user in domains such as retail and airlines, calls domain-specific APIs, follows policy guidelines, and ends with a database state that is compared against an annotated goal state; alongside one-shot success, the benchmark measures repeated reliability with $\text{pass}^k$, and retail $\text{pass}^8$ is below 25% (Yao et al., 2025). A single success shows that the agent found one viable path, whereas a team needs sufficient assurance before it can delegate the same class of task reliably. Generation volume can grow with models and compute, while confidence is constrained by test coverage, CI, staged rollout, audit, rollback capacity, and the number of people who have time to review. Because generation capacity and trusted assurance capacity can grow at different rates, assurance can become the binding constraint on an organization’s ability to expand delegation (Figure 3).
If a person has to inspect every step, tool call, and diff with equal intensity, more autonomous agents create more work for people. The engineering approach to reducing this cost is the runtime expression of supervision leverage: OpenAI’s account of building an agent-first software product treats human time and attention as scarce resources, placing more domain context, architectural constraints, observability, and verification mechanisms into the repository and harness so that agents can operate over a larger problem space (Lopopolo, 2026). People spend less effort producing and checking code line by line, and more effort constructing an environment in which code can be generated, checked, and recovered reliably, work that also disappears from the cost accounting of any single delegation.
As compilers, linters, and CI suites filter out obvious breakage automatically, what reaches a human reviewer changes shape. The outputs that land on a person’s desk are typically the ones that look right on the surface: the code compiles, the tests pass, and the agent reports completion. Reviewing work that already passes automated checks remains demanding because the mechanical errors have already been stripped away, leaving three judgments that automated tests cannot settle:
- Whether the result matches what was actually wanted. Automated checks verify compliance with a written test or specification. If the agent wrote the tests itself under a flawed understanding of the requirement, implementation and test may drift together. Even with an independent test suite, inspecting the concrete patch is often the moment when the user realizes that an unstated preference was missed or that a technically valid solution solved the wrong problem.
- Whether the implementation is worth keeping in the system. A patch can pass functional checks while introducing brittle structure or omitting operational basics that make future maintenance harder. A study of 4,550 agentic pull requests across 81 open-source repositories found that agents handle logging differently from human developers, rarely adding sufficient logging on their own, and that humans ultimately performed 72.5% of post-generation log repairs, acting as what the authors call “silent janitors” who clean up observability problems left by agents even when review feedback never mentioned the requirement (Ouatiti et al., 2026). The study measures who repairs logs; reading it as “the initiator receives the benefit while downstream maintainers absorb the cost” is my inference, though it matches the experience of many teams. That is why HM/VVO sums over $r$: at least three roles may sit inside $R_i$, namely the requester who initiates the delegation, the reviewer who checks it, and the downstream maintainer who inherits it. Counting only the requester’s time and stopping at delivery is the accounting choice that hides the downstream cost.
- Whether the humans responsible for the system still understand how it works. In any codebase or workflow that outlives a single prompt, someone has to debug failures, extend functionality, and take responsibility when incidents occur. When reviewers treat a green test run as sufficient to merge large changes they have not digested, the team’s grasp of its own system erodes. Reviewing a substantive change serves both to catch defects and to help the human owners absorb what changed so they remain able to maintain the system afterward.
Which Loop Should the Human Be On?
If step-by-step babysitting wastes attention and waiting until the very end risks reviewing a large, test-passing artifact built in the wrong direction, where should human judgment actually sit? Many products still implement “human in the loop” either as an approval prompt before every tool call or as a single sign-off at the end. Both designs treat human attention as uniform. In practice, people enter an agent’s trajectory for four distinct reasons (Figure 4): low-risk, reversible actions backed by strong automated checks can proceed directly; choices that hinge on unstated user preferences or private context warrant a clarifying interaction; actions that touch money, permissions, production state, security, or other irreversible boundaries require explicit authorization; and runs where verifiers conflict, state looks corrupted, or the agent cannot diagnose a failure call for escalation and human takeover.
Interrupting the user whenever the model is uncertain is a poor default, because every question taxes human attention and most answers do not change what the agent should do next. A more disciplined criterion weighs the value of information against the cognitive cost of interrupting the user, taking into account task risk, ambiguity, and the effort required to answer (Dong et al., 2026). That trade-off explains why a chatty agent can feel just as exhausting as a reckless one. Deciding when to ask is a distinct skill that does not automatically improve alongside raw problem solving: on HiL-Bench, which hides missing, ambiguous, or contradictory requirements inside software engineering and text-to-SQL tasks so they surface only during exploration, models that perform well with complete specifications recover only part of their score when they must decide on their own when to call ask_human() (Elfeki et al., 2026).
Verbal questions are also only one way to get human input, and often an inefficient one when the user needs to see something concrete to know what they want. Putting a cheap prototype, a reversible edit, a draft pull request, or a dry-run preview in front of the user early in the trajectory can resolve refine and reconstruct far faster than an abstract questionnaire. Showing an early, low-cost artifact also directly reduces the post-completion review burden described in the previous section: catching a framing mistake on a fifty-line sketch spares the reviewer from untangling a two-thousand-line implementation that already passes all its tests. Because this tactic relies on being able to throw the draft away cleanly, reversibility acts both as a safety boundary and as a design lever that lets concrete previews replace rounds of questioning.
Where irreversible actions are concerned, capability gains do not erase the boundary between prediction and permission. A model may grow arbitrarily good at predicting which actions a user would approve, yet predicting an approval and holding the authority to act remain different things. At the same time, requiring a human to authorize an action does not require making them click a button on every step. Defining explicit scopes of authority in advance, specifying which classes of actions may run automatically, under what preconditions, and within what blast radius, lets routine work proceed uninterrupted while reserving live approval for actions that cross the boundary.
Experienced agent users already allocate their attention in this selective way. Telemetry across millions of human-agent interactions in Claude Code and Anthropic’s public API shows that more experienced users enable auto-approval more often while also interrupting the agent more frequently (Anthropic, 2026). For these users, autonomy and oversight do not sit at opposite ends of a single slider: seasoned users stop clicking routine permission prompts and intervene more actively when something matters. Although the telemetry does not record the exact reason for each interruption, the pattern is consistent with users shifting their effort away from mechanical step-by-step confirmation toward high-leverage course corrections. Systems such as Magentic-UI build these distinct entry points directly into the interface through upfront co-planning, live co-tasking and browser takeover, action guards, and answer verification (Mozannar et al., 2025).
Concentrating human involvement into fewer, higher-stakes moments carries a structural trap of its own. Across 106 studies and 370 effect sizes, a meta-analysis of human-AI collaboration found that combined human-AI teams performed worse on average than the stronger of the human or the AI working alone, particularly on decision-making tasks, with positive gains depending heavily on how tasks and information were divided (Vaccaro et al., 2024). Simply inserting a person into the workflow guarantees nothing. Worse, as Lisanne Bainbridge observed in her classic study of industrial and aviation automation, turning human operators into passive monitors of a mostly reliable system erodes the very hands-on familiarity and skill they need when forced to step in during rare, complex failures (Bainbridge, 1983). This connects directly back to why human reviewers have to digest the changes an agent makes: if automation handles routine runs and humans only glance at green checkmarks, the people nominally in charge steadily lose their mental model of the system, leaving them ill-equipped to judge the hardest edge cases or recover when an autonomous run goes off the rails.
The Human Is Part of the Agent
Step back from these individual stages, and a consistent pattern emerges: the conditions that determine whether a delegation succeeds mostly sit outside the software box we call “the agent.” A benchmark score omits the human judgment spent crafting the task and its verifier. A product metric that logs “task completed” omits the time a reviewer spends reading the diff and the time a maintainer spends cleaning up logs months later. The agent runtime itself does not decide who holds organizational authority or who answers for a production incident. From the perspective of the person typing the prompt, the teammates who absorb the downstream review and maintenance work may be out of sight as well. In every case, the system that actually produces a working, trusted result is larger than the agent alone.
Rapid gains in model capability can create the illusion that human involvement is simply vanishing, when much of it is changing position. The evolution of training and evaluation already showed how sample-level grading gave way to writing principles, building verifiers, and designing environments. A parallel shift is now playing out at runtime. As models and harnesses get better at execution control, including remembering constraints across long contexts, checking workspace state, gathering evidence before editing, running tests, and rolling back failed attempts, much of today’s babysitting will fade away. It should fade away; compensating for brittle execution control is unfinished engineering rather than a lasting human role.
Once that operational babysitting recedes, human work separates into two distinct locations: what can be specified in advance moves upstream before execution begins, while the judgments that depend on inspecting a concrete artifact remain during the run and at delivery.
Moving upstream before runtime are the repeatable rules of the game: architectural constraints, coding standards, automated test suites, sandbox boundaries, predefined scopes of authority, and clear lines of organizational accountability across operators, engineering owners, managers, vendors, and regulators. Putting those guardrails in place ahead of time removes the need for repetitive per-action clicks and ensures that when an incident occurs, liability does not fall by default on whoever happened to press an Approve button last.
Remaining during the trajectory and at final review are the tasks that no upfront rule or automated test suite can finish on its own:
- Forming the rest of the goal through concrete artifacts. Because people often discover what they want and what trade-offs they accept only after seeing a working draft, agents need to surface low-cost, reversible previews at key decision forks so direction can be corrected before thousands of lines of polished work accumulate.
- Reviewing work that already looks right on the surface. As automated checks catch syntax errors, broken builds, and obvious regressions before a human ever looks, human review concentrates on what passes the test suite: verifying that the solution addresses the real underlying intent, judging whether the code or artifact is clean and maintainable enough for downstream teammates to inherit, and understanding the change well enough to own and debug it in production.
- Reconstructing context when an autonomous run escalates. The longer an agent runs without routine human involvement, the less situational awareness the human retains when an anomaly occurs or a complex deliverable lands for sign-off. As routine interruptions decline, the central bottleneck shifts from whether people are willing to step in to whether they can understand the situation in time. That makes the quality of the handoff a core design problem for agent products: presenting the relevant history, current state, what automated checks have already verified, the open trade-offs, downstream impact, reversibility, and required authority in a form a human can actually evaluate, where most tools today still offer little more than a bare
AlloworDenyprompt.
Evaluations are gradually expanding to reflect this wider boundary. $\tau$-bench asks whether a workflow is reliable across repeated runs; HiL-Bench tests whether an agent knows when to pause and ask; SWE-Together counts how many rounds of human course correction a trajectory takes. Each of these benchmarks runs into the same measurement challenge that teams face in production: once the human is recognized as part of the system, what we are evaluating and designing is the joint machinery through which human judgment and model execution turn an initial intent into a result someone can understand, trust, and stand behind.