The Human Is Part of the Agent

Over the past few years, we have grown accustomed to chasing model capability with ever harder benchmarks. When MMLU stopped being hard enough, we got GPQA. As competition mathematics approached saturation, evaluations moved to harder AIME and IMO problems, then to Humanity’s Last Exam. Coding benchmarks progressed from HumanEval to SWE-bench, Terminal-Bench, and real software engineering tasks that unfold over much longer time horizons. The chase itself is getting harder. The Stanford 2026 AI Index notes that some benchmarks intended to last for years began approaching saturation within months. Humanity’s Last Exam was designed specifically for frontier models and deliberately weighted toward questions that challenge human experts, yet its top score still rose by roughly 30 percentage points in a single year. We keep extending the ruler because each old one becomes too short more quickly than the last (Stanford Institute for Human-Centered Artificial Intelligence, 2026). This is reasonable. A benchmark uses a set of repeatable, objectively scored problems to approximate something that is otherwise difficult to define: how smart a model is.

As benchmarks become harder, model capability is reaching more intuitive human reference points: competition medals, domain experts, and genuinely unsolved research problems. In July 2025, OpenAI’s general-purpose reasoning model scored 35 out of 42 on the International Mathematical Olympiad, reaching gold-medal level. By February 2026, experts judged at least five of ten proof attempts in First Proof likely to be correct. Some of those problems had remained open for years before their authors found solutions (OpenAI, 2026). The curve continued upward. Ten results announced in August 2026 each solved or substantially advanced a longstanding open problem in mathematics or theoretical computer science (OpenAI, 2026). These results came with different conditions, including multiple attempts, human selection, and expert feedback. It would be inaccurate to read every one as an independent model achievement. The direction, however, is clear. Mathematics is a relatively clean example. In cybersecurity, the practical consequences of the same capability curve are more immediate. Mythos Preview can discover and exploit zero-day vulnerabilities in major operating systems and browsers, and many previously unknown vulnerabilities were found autonomously after the initial prompt (Anthropic, 2026). Because this capability is asymmetric between offense and defense, programs such as Project Glasswing give defenders earlier access (Anthropic, 2026).

If we follow only this curve, an obvious intuition emerges: agents have become extremely smart. They can solve mathematical problems beyond the reach of most people, work at a frontier accessible to only a small number of domain experts, and outperform human experts on some highly specialized tasks. For many frontier tasks, an ordinary person’s position has shifted from “Would I do better than the model?” to “I may not even understand what it is doing.” If smartness means the ability to solve increasingly difficult problems, the past few years have been a direct exercise in smartness maxing, with little sign of slowing down. Yet this smartness does not fully carry over into the experience of using an agent. After working daily with Claude Code, Codex, or any sufficiently capable agent, you may arrive at a subtler judgment: the model is unquestionably powerful, sometimes absurdly so, yet on tasks that appear far easier than IMO problems, research mathematics, or vulnerability research, it can still feel strangely stupid. The feeling is hard to pin down. The task may be complete, the code may run, and the answer may contain no identifiable factual error, yet something still feels wrong. Switch models, change the interaction style, or add a few rounds of back-and-forth, and the sense of whether the agent is pleasant to use can change dramatically. This quality does not collapse into a number as neatly as benchmark accuracy, yet people who use agents regularly tend to recognize it quickly.

The problem is that the word “smart” is being asked to evaluate several different properties of a system. A benchmark primarily answers one question: given a fully specified problem, how difficult a problem can the solver handle? That is problem-solving capability. Real use exposes four more questions. Can the agent apply its capability reliably across a long trajectory? That is agentic control. Can the system continue forming and revising the task through interaction? That is task formation. Is there sufficiently independent evidence to justify adopting the result? That is assurance. Who authorizes an action, and how is responsibility assigned when something goes wrong? That is governance. All five can overlap in a single failure, and they improve at different rates (Figure 1). Compress them into one judgment about whether the agent is smart, and benchmark scores can keep rising while day-to-day usability fails to improve in lockstep.

What, Exactly, Feels Stupid?

This kind of “stupidity” can be described quite precisely. In real use, people rarely complain that a model lacks reasoning ability. They are more likely to say, “You knew that constraint. Why did you violate it again?”, “You started changing things without confirming anything,” “I just changed that. Didn’t you notice?”, “You said it was done without verifying it,” or “You made a mess and can’t roll it back.” Every complaint assumes that the model was capable of doing the right thing. The failure lies in how it managed that capability along an unfolding trajectory. Translating these complaints into more system-oriented language shows that they arise at different layers and require different remedies:

The first eight rows mostly concern agentic control, which training and the harness must address together. Constraints and plans may disappear from the context after dozens of steps, or they may remain present without being followed. Most failures involve some combination of both, so I will not try to assign every row exclusively to one side. The ninth row has already moved into assurance. “You wrote the tests and then used them to validate your own work” is my own judgment rather than a claim directly established in the literature. If the agent has misunderstood the requirement, its tests may drift in the same direction as its implementation; a green test run then provides no independent evidence. The next two rows concern task formation: one involves information that has not yet been obtained, and the other a judgment that has not yet been formed. The final row concerns governance, which determines which actions require approval and from whom. These functions overlap. The table identifies the first problem each complaint reveals.

Most of the evidence that follows comes from coding agents. Software engineering provides executable tests, git, diffs, CI, sandboxes, and rollback, which make failures unusually visible, localizable, and countable. The tradeoff is that software engineering differs sharply from legal, medical, financial, and organizational decisions, where correctness may itself be ambiguous and reversal difficult. The evidence below is useful for identifying mechanisms. Its measured proportions should not be generalized directly to every industry.

The scale of these complaints is substantial. An observational analysis of 20,574 coding-agent sessions across 1,639 repositories identified seven recurring forms of misalignment, spanning project understanding, intent interpretation, rule adherence, action boundaries, code execution, and progress reporting. In 90.5% of episodes, the failure imposed a cost on effort or trust. Most sessions did not confirm the final outcome in the visible conversation, so resolution could be observed in only 9.33% of episodes, or 1,504 cases. Among those, 91.49% required explicit user correction (Tang et al., 2026). The same data also showed a compositional shift: as the overall problem rate declined, constraint violations and inaccurate self-reports made up a larger share of what remained. Observational data cannot establish that stronger models caused this shift. It does suggest that the remaining problems increasingly fall outside a simple inability to perform the task.

Two forms are especially easy to reproduce. The first is failing to stop. All 200 FixedBench tasks come from issues that had already been fixed, so the correct action is to recognize that fact and leave the code unchanged. Across five recent models and four harnesses, agents still proposed unnecessary changes in 35% to 65% of cases. Requiring them to reproduce the problem before patching reduced some over-action, while creating under-action on issues that had been only partially fixed (Gloaguen et al., 2026). FixedBench does not establish why agents are so inclined to act. One plausible explanation is that, in most training and evaluation signals they have seen, doing nothing is rarely counted as success. The second form is failing to notice that the world has changed. In real development, users read code, edit it, and sometimes revert changes while an agent is working. The repository state observed by the agent is therefore jointly produced. When simulated user edits relevant to the task and capable of changing the completion path were injected into SWE-bench Verified, the average resolve rate fell by 7.7 percentage points. Trajectory analysis showed agents retaining code that conflicted with user edits or overwriting those edits without rechecking the workspace or running targeted tests (Tan et al., 2026). Standard SWE-bench cannot expose this behavior because nobody else changes its environment, leaving no shared state to reconcile.

Final outcomes also reveal little about where an agent first went off course. Human annotation of 115 failed trajectories identified nine failure categories, including plan-adherence failure, invalid invocation, misreading tool output, intent-plan misalignment, and underspecified user intent. The annotators also tried to locate the first unrecoverable critical step in each trajectory. These trajectories came from three distinct domains: structured API workflows, incident management, and open-ended web and file tasks (Barke et al., 2026). The same failure modes recur across long trajectories even when the task has nothing to do with writing code.

Calling these failures “the model cannot do the task” has become increasingly misleading. The model is usually capable of completing each local step. The open question is whether, after dozens or hundreds of steps, it can still honor the original constraints, maintain current state, gather evidence before acting when evidence is weak, control scope, know when to stop, verify the outcome, report it honestly, and recover after a mistake. I will call this agentic control. It is distinct from the difficulty of the problems a model can solve, and it is the object of many complaints that begin, “You clearly know how, so why did you do that?” I expect this layer to improve substantially because each component offers concrete engineering levers: persistent constraints, refreshed workspace state, pre-action evidence thresholds, automated verification, checkpoints, and rollback can all be addressed through the harness, context management, and training incentives. This is a prediction without long-term evidence. FixedBench’s under-action tradeoff also shows that reducing one control failure can produce another. A more defensible expectation is that a meaningful share of today’s perceived stupidity will decline as systems improve. One question remains: if control failures are so common in real use, why are they nearly invisible on benchmarks?

What Benchmarks Fix for the Agent

Benchmarks largely miss these problems by design, and that design is appropriate for what they measure. SWE-bench gives an agent a real GitHub repository, an issue description, and a code environment, then uses tests to determine whether the resulting patch works (Jimenez et al., 2023). SWE-Lancer extends the same idea to more than 1,400 tasks drawn from real freelance work and maps them to roughly one million dollars in actual payouts (Miserendino et al., 2025). Both answer a solver question: given a written task and a fixed success criterion, how difficult a problem can the model solve? Meaningful measurement requires the task, environment, and verifier to remain fixed. In real delegation, a larger system must continually supply and maintain those conditions. The work may fall to the agent, harness, user, CI, or organizational process. The task may be reinterpreted, other people may modify the environment concurrently, success may depend on preferences the user has not articulated, and validation may come from an independent checker or reviewer. A benchmark fixes these variables on behalf of the whole system for the duration of a run. A component capability score therefore cannot predict how the same model will perform in a real delegation.

Preparing those conditions is work that people have already done, and it is expensive. DeepSWE designed 113 long-horizon software engineering tasks from scratch across 91 real repositories and hand-wrote a behavior-oriented verifier for each task. These verifiers accept implementations the researchers had never seen as long as they satisfy the task, placing judgment on the requested functionality. Independent LLM judges disagreed with them only 1.4% of the time, compared with 32.4% disagreement on tests inherited from SWE-Bench Pro (Huang et al., 2026). The scarce human judgment here sits upstream of “solve these 113 tasks first.” It determines which tasks are worth measuring, which complexities of a real environment should remain, and what an entirely unfamiliar solution must satisfy to count as success.

This shift upstream began before modern agent evaluation. In early RLHF, labelers wrote ideal responses to prompts submitted by real users for supervised fine-tuning, ranked multiple model-generated candidates, trained a reward model on those rankings, and used it to continue optimizing the policy (Ouyang et al., 2022). Most human effort was spent on individual samples. Models later absorbed much of this work. In 2023, ChatGPT outperformed crowdworkers on four of five text-annotation task categories, achieved higher inter-coder agreement, and cost roughly one twentieth as much per item as MTurk (Gilardi et al., 2023). GPT-4 acting as a judge agreed with human preferences more than 80% of the time, with systematic costs such as position and verbosity bias (Zheng et al., 2023). Much of the work being absorbed consisted of bounded judgments attached to single samples, allowing more human effort to move toward judgments with broader coverage. Constitutional AI makes this shift explicit. Humans write a set of principles; the model uses the constitution to critique a response, identify violations, and revise it. The same principles can then compare responses and generate preference data for reinforcement learning (Bai & others, 2022). A human no longer has to judge every response individually. One principle can constrain an entire class of future cases.

Across these different research threads, a common dimension emerges: how much future behavior a single human judgment can constrain. I will call this dimension supervision leverage (Figure 2):

Toward the left, the human resembles a producer of answers for the model. Toward the right, the human defines the problem space in which the model should operate and what success means inside it. A label affects one sample; a principle constrains a class of outputs; a verifier defines a set of acceptable solutions; the task and environment establish the boundary of the entire problem space. This spectrum describes judgment coverage. By itself, it says nothing about whether total human hours rise or fall. It supports a narrower conclusion: some sample-level labor is declining, while some judgment is moving into principles, verifiers, task design, and environment design. These new positions often fall outside the scoring boundary. A SWE-bench score does not account for the judgment spent designing its environment, just as a successful delegation does not account for all the context a user supplied to make it work. This displacement will appear several more times, becoming harder to see each time.

This progression rests on another assumption: the task, rubric, and objective can be fully specified before the agent begins. Benchmarks turn that assumption into a design condition, which is precisely how they measure solver capability reliably. User-facing agents rarely receive such clean conditions.

We May Have Been Evaluating a User Who Does Not Exist

These evaluations contain an unusually cooperative user. This user can state the full requirement at once, provide all relevant context, define the success criterion, and hold the same judgment throughout the task. They speak once at the beginning; tests, a verifier, or a judge determine correctness afterward. Isolating problem-solving capability requires this kind of user. Real work rarely provides one.

An engineer might say, “This feature has become too complicated. Simplify it.” They may be unable to say how simple is simple enough. After seeing the first revision, they realize that their actual concern is whether a new teammate can understand the code, and that some apparently redundant branches encode lessons from earlier production incidents. The original request was valid, yet it needed a concrete alternative before it could become specific enough. A behavioral analysis of 11,579 AI-assisted IDE sessions found the same pattern. Developers rarely state everything at once. They tighten the objective through a sequence of short prompts and change direction after seeing intermediate results (Tang et al., 2026).

Three different processes are mixed together here. Sometimes the user knows what matters from the start and the agent simply lacks that information. The agent needs to ask; this is reveal. Sometimes the user has only a direction and needs to see an implementation and its tradeoffs before expressing the requirement precisely. The agent must first produce something concrete enough to evaluate; this is refine. In the third case, the agent’s action changes how the user understands the problem and creates a judgment that did not exist when the interaction began. The artifact itself helps the user think; this is reconstruct. All three make the initial prompt appear incomplete, yet they call for different behavior. Reveal requires the agent to ask. Refine requires it to act first and produce something evaluable. Reconstruct means that part of the delegation’s objective is created during the delegation itself.

Stronger models can reduce much of the cost of the first two processes. They can infer more latent preferences from context, store stable preferences in memory, and distinguish uncertainties that would change the next action from those that would not. Ambig-SWE measures this gap directly. It converts SWE-Bench Verified into deliberately underspecified tasks, requiring an agent to notice that the instruction lacks critical information and obtain it through targeted clarification. Models do not reliably know when to ask. Allowing interaction in this setting improved performance by as much as 74% relative to the non-interactive setting (Vijayvargiya et al., 2026). This gap can be trained, evaluated, and engineered. The third process cannot be eliminated through ex ante preference inference. If a judgment forms only after concrete consequences become visible, an earlier question cannot retrieve it. Cheap prototypes, reversible actions, and quick presentations of alternatives can lower the cost of forming the judgment, while the interaction itself still has to occur. A user profile created before execution is therefore insufficient. Preferences continue to change before and after action, making personalization an iterative loop (Liang et al., 2026). Compressing different users’ utilities into one population-level reward can also produce an objective that represents none of them particularly well (Poddar et al., 2024).

Evaluation is changing accordingly. SWE-Together selected 109 repository-level tasks from 11,260 real user-agent coding sessions. A reactive user simulator preserves the original user’s intent and sequence of interventions, while the evaluation records both final repository correctness and the number of corrective-feedback rounds required along the way (Wu et al., 2026). The object of measurement expands from independent task completion to whether an agent can carry a trajectory forward under user clarification, constraint, and correction. Interactive evaluation necessarily relies on a user model, and the fidelity of that model changes what the score means. A comparison involving 31 user simulators, 451 real participants, and 165 tasks found that LLM simulators were overly cooperative and stylistically narrow. They lacked the frustration and ambiguity of real users, gave consistently more positive feedback, and created an easy mode that inflated agent performance. Stronger models were not automatically more faithful simulators (Zhou et al., 2026). Once users enter the evaluation, the kind of user being modeled becomes an experimental parameter that requires calibration. Solver benchmarks do not face this parameter at all.

What is missing here belongs to a different problem from agentic control. Even with perfect constraint following, state tracking, and verification, a user who speaks only at the beginning will still underspecify the task. They possess state the agent cannot see and continue supplying information that determines what the task has become across the trajectory. Interaction corrects errors and helps form the objective. Final pass rate alone omits the human effort required to sustain that interaction, which leads to the cost of delegation itself.

Where the Cost of Delegation Goes

If AI cuts the time spent writing code in half, an engineer’s total time in real projects should fall substantially. The data are less tidy, and the gap appears outside generation. The gain in generation is real. On a well-bounded programming task with automatically executable tests, the treatment group using GitHub Copilot finished 55.8% faster on average than the control group (Peng et al., 2023). The experiment also reveals the scope of that result: gains in generation speed are easiest to measure when the task, context, and success criterion have already been fixed.

The result reversed in mature repositories familiar to the developers. In 2025, METR asked 16 experienced open-source developers to complete 246 real issues in their own repositories. With the AI tools available at the time, completion took roughly 19% longer on average. Before the study, participants expected to be 24% faster; afterward, they still believed they had been about 20% faster (METR, 2025). The distance among these three figures is itself a result: in this experiment, participants’ sense of the time they spent diverged from the measured time. METR’s later experiment produced a speedup in the raw estimates, although the confidence interval crossed zero and the participant and task mix had shifted substantially. METR therefore described the data as an unreliable signal of the current productivity effect and treated it only as weak evidence that tools may have become more helpful since 2025 (METR, 2026). Neither study can establish the overall direction of AI’s effect on developer productivity by itself. Together they make a narrower point: several costs outside generation remain easy to omit.

We can write those costs explicitly:

\[\text{Net Delegation Value} = U_{\text{verified outcome}} -\lambda_h C_{\text{human effort}} -\lambda_t C_{\text{latency}} -\lambda_c C_{\text{compute}} -\lambda_r E[\text{risk}]\]

$U_{\text{verified outcome}}$ is an outcome the user will accept, use, and treat as grounds for delegating again. $C_{\text{human effort}}$ includes prompting, context setup, monitoring, intervention, review, correction, and rework. The remaining terms capture waiting, compute, and risk. This is a conceptual equation. It reminds us that an improvement in autonomy, pass rate, or latency moves only one term on the right-hand side, so net value may not improve with it. For a set of delegations with similar task value and verification thresholds, I would also examine the ratio between the first two terms: how many minutes all participants spend to obtain one outcome that genuinely counts. I will call this Human Minutes per Verified Valuable Outcome (HM/VVO):

\[\text{HM/VVO} = \frac{\sum_{i=1}^{N}\sum_{r\in R_i} m_{ir}}{\sum_{i=1}^{N} s_i}\]

$N$ is the number of delegations during the period, $R_i$ is the set of all roles involved in delegation $i$, and $m_{ir}$ is the number of minutes role $r$ spends on it. Understanding, clarification, oversight, verification, maintenance, and rework all count. $s_i$ equals 1 only when the outcome both passes verification and is actually adopted.

Three easy-to-miss conditions must remain visible in this definition. First, the verification threshold for $s_i$ must scale with risk. A disposable script and an operation that changes production permissions require entirely different levels of confidence. Verification for the latter may cost more than generation itself. Before comparing systems, the threshold must therefore be held constant; otherwise, reducing review makes the metric look better without producing an equally trustworthy outcome. Second, task values cannot vary too widely. If a low-value edit and a critical migration each count as one outcome, the ratio loses meaning. Third, the numerator must include everyone’s time. That condition will matter immediately.

Decomposing $C_{\text{human effort}}$ one level further reveals three qualitatively different costs. One is spent steering the agent in the right direction: explaining the goal, supplying context, and asking or answering clarifying questions. Another is spent watching the process: seeing what the agent is doing, noticing drift, interrupting, correcting, and recovering. A third is spent after completion establishing confidence in the result: reading the diff, running tests, checking edge cases, and confirming that the agent did not change anything outside scope. Each cost requires a different remedy. Collapse them into “agents are tiring to use,” and the system gives no clue about what to improve.

Early clarification primarily reduces misunderstanding, correction, and rework. It may leave the need for independent assurance unchanged. In a health-information setting, a randomized, blinded study rated Wayfinding AI, which actively requests context, as more helpful, relevant, and tailored than the baseline. Users were also willing to wait longer in exchange for providing more context (Sayres et al., 2025). The study measured users’ evaluations of the answers and their willingness to wait. It did not directly measure the exchange rate between one more early question and one less later revision. When an agent asks the wrong question, the question itself is pure overhead.

The shape of the second cost depends on whether the user can understand what the agent is doing. Interviews with 17 experienced developers divided oversight into a priori control, co-planning, real-time monitoring, and post-hoc review. They also documented heuristics such as running tests or scanning a diff to reduce the amount of line-by-line reading (Dhanorkar et al., 2026). These heuristics save attention only when their evidence actually supports continuing, rather than merely showing that the agent completed the local objective it defined for itself. Without deliberate design, these costs surface together as coordination overhead. A study in which 31 participants used commercial agents on representative tasks recorded mental-model mismatch, failure to establish trust, incompatible collaboration styles, and excessive communication overhead (Shome et al., 2025). These difficulties fall into two broad groups: how users communicate goals, preferences, and constraints to agents, and how agents communicate their capabilities, plans, actions, and side effects to users (Bansal et al., 2024). Without common ground, users cannot tell whether they have been understood or when they should intervene.

The third cost is usually the hardest to compress. In $\tau$-bench, an agent converses with a simulated user in domains such as retail and airlines, calls domain-specific APIs, follows policy guidelines, and ends with a database state that is compared against an annotated goal state. Alongside one-shot success, the benchmark measures repeated reliability with $\text{pass}^k$; retail $\text{pass}^8$ is below 25% (Yao et al., 2025). A single success shows that the agent found one viable path. A team needs sufficient assurance before it can delegate the same class of task reliably. Generation volume can grow with models and compute, while confidence is constrained by a different set of conditions: test coverage, CI, staged rollout, audit, rollback capacity, and the number of people who have time to review. Generation capacity and trusted assurance capacity can therefore grow at different rates, making assurance the binding constraint on an organization’s ability to expand delegation (Figure 3). I use assurance here because confidence can be established through tests, independent checkers, rollout, audit, and human review. Line-by-line reading is only the most expensive route.

If a person has to inspect every step, tool call, and diff with equal intensity, more autonomous agents create more work for people. The engineering approach to reducing this cost is the runtime expression of supervision leverage. OpenAI’s account of building an agent-first software product treats human time and attention as scarce resources. It places more domain context, architectural constraints, observability, and verification mechanisms into the repository and harness so that agents can operate over a larger problem space (Lopopolo, 2026). People spend less effort producing and checking code line by line, and more effort constructing an environment in which code can be generated, checked, and recovered reliably. That work also disappears from the cost accounting of any individual delegation.

There is a further complication: the person initiating a delegation may not be the person who pays these costs. A study of 4,550 agentic pull requests across 81 open-source repositories found that agents handle logging differently from human developers. They rarely add sufficient logging unless explicitly asked, and humans ultimately performed 72.5% of post-generation log repairs. The authors call these people silent janitors because they clean up observability problems left by agents even when review feedback never mentioned the requirement (Ouatiti et al., 2026). The study measures who repairs logs. Reading it as “the initiator receives the benefit while downstream maintainers absorb the cost” is my inference, although it matches the experience of many teams. This is why HM/VVO sums over $r$. At least three people may sit inside $R_i$: the requester who initiates the delegation, the reviewer who checks it, and the downstream maintainer who inherits it. Choosing whose time to include is itself a decision. The easiest accounting choice, counting only the requester’s time and stopping at delivery, is precisely the choice that hides the cost.

Which Loop Should the Human Be On?

“Human in the loop” is often implemented as an approval button before every action or as one final confirmation. Both designs treat user attention as if it can be spread uniformly across a trajectory. Clarification, oversight, and verification have different costs and return different information. If human involvement is allocated by information value, people enter a trajectory in roughly four ways. Low-risk, reversible actions with strong machine verification can be executed directly by the agent. Choices that depend on user-only information or preferences justify clarification. Actions involving money, permissions, production state, security, or other irreversible consequences require explicit approval. When verifiers disagree, state becomes anomalous, or the model cannot explain the cause of a failure, the system should escalate. The distinction is what the person is being asked to contribute: information, judgment, authorization, or takeover after the agent has exhausted its ability to proceed (Figure 4).

“Ask a human whenever uncertainty exists” is not a workable policy. Questions consume user attention, and most answers would not change the agent’s next action. A better criterion is the value of information: compare the expected benefit of possible answers with the user’s cognitive cost. The result depends on task risk, ambiguity, and the effort required from the user (Dong et al., 2026). This explains why an agent that asks more questions may still be worse to use. Knowing when to ask is a capability of its own and does not arrive automatically with greater problem-solving capability. HiL-Bench adds missing, ambiguous, or contradictory information to software engineering and text-to-SQL tasks, with the blocker revealed only through progressive exploration. The agent must identify the blocker, call ask_human(), and ask a targeted question. Ask-F1 measures both question precision and blocker recall. Even models that perform well with full information recover only part of that performance when they must decide for themselves whether to seek help (Elfeki et al., 2026).

Questions are only one way to obtain human judgment, and sometimes a costly one. When a preference becomes clear only after the user sees a concrete implementation, a reversible edit, draft pull request, or dry run may reveal what they want faster than a series of abstract questions. These actions serve as judgment aids by creating the object needed for refine and reconstruct. Their cost is that they must be genuinely reversible. Reversibility is therefore both a safety constraint and a condition that determines how often an agent can substitute action for a question.

One boundary will remain as capability improves. An agent may become increasingly accurate at predicting what a user would approve. Prediction describes what is likely to happen; authorization determines what is allowed to happen. Better prediction resolves only the first question.

“Someone must permit this” and “someone must click every time” are separate requirements. Most per-action approvals can be replaced by scopes of authority defined in advance. Specify which actions may proceed directly, under what conditions, and within what impact radius. The agent can then avoid returning for permission at every step, while the user saves attention for consequential decisions. What remains is the authority to draw that scope and the actions outside it whose impact is large enough to require direct approval.

Under these constraints, mature usage looks somewhat counterintuitive. An analysis of millions of human-agent interactions from Claude Code and the public API found that experienced users auto-approve more often and also interrupt more often (Anthropic, 2026). For these users, autonomy and oversight do not trade off along a single axis. They stop clicking every permission prompt and interrupt more frequently. The data do not show when those interruptions occur. My interpretation is that experienced users move confirmation away from every step and toward the moments they consider worth managing. They are choosing where to spend attention. Magentic-UI builds that choice into the interface and agent architecture. Co-planning lets users change the plan before execution; co-tasking lets them provide natural-language feedback or take over the browser during execution. The system also provides action guards, answer verification, memory, and multitasking (Mozannar et al., 2025). Supplying objective information, modifying a plan, taking over a local action, and verifying only the final result are distinct positions in a delegation workflow, rather than opposite ends of one switch.

Putting a person back into the loop does not automatically improve the outcome. A meta-analysis of 106 experiments and 370 effect sizes found that human-AI combinations performed worse on average than the stronger of the human or AI alone. Performance losses were especially common in decision tasks, and gains depended on design factors such as task allocation and information flow (Vaccaro et al., 2024). Participation alone creates no benefit. An older warning still applies: automation turns operators from performers into monitors, even though people are poor at watching a system that is almost always correct for long periods. As automation takes over routine situations, operators lose practice and their skills decay, yet they are expected to intervene in the rarest and hardest cases (Bainbridge, 1983). Together, these findings show a cost to concentrating human judgment at a few high-value moments. With fewer interventions, people get less practice, while the cases that reach them are precisely the difficult ones.

Putting the human on the right loop is therefore a design objective. It will not emerge automatically as a byproduct of capability gains. Human judgment can concentrate on goals, context, exceptions, verification, approval, and accountability only when the system prepares sufficient context, recoverability, and means of maintaining human understanding for those moments. Without them, greater autonomy leaves people facing harder problems at fewer points in time.

The Human Is Part of the Agent

Many conditions that determine whether a delegation succeeds lie outside the box we habitually label “the agent.” A SWE-bench score excludes the judgment of the people who designed its task and verifier. A product metric called “completion” excludes later review, maintenance, and recovery. The software object called an agent excludes who can approve an action and who owns the system. From the requester’s perspective, the other people affected by the delegation may also be invisible. These omissions take different forms and point in the same direction: the system that actually gets the work done is larger than the box we draw.

Capability growth can obscure this point because work moved outside the box can look as though it has disappeared. The spectrum from labels to task design has already shown this once. People write fewer answers sample by sample, while judgment moves into selecting tasks, writing verifiers, and deciding whether an unfamiliar solution counts as success. A second shift is underway. After absorbing execution, agents are beginning to absorb routine control: remembering constraints stated last week, refreshing state, gathering evidence before acting, running verification, and rolling back after errors. They should absorb this work. Maintaining an agent’s basic control is unfinished engineering and is unlikely to become a durable human role.

Once that work is absorbed, three qualitatively different categories remain for people: objective formation handles information and preferences that have not yet settled; assurance handles the evidence required to trust a result; authorization and accountability determine what may happen and who bears responsibility.

First, the objective is still forming. Part of a task becomes expressible only after the agent produces something concrete. Agents can improve at asking questions and can present reversible versions for users to inspect, yet the content elicited by those interactions must come from people. This work will change form rather than disappear.

Second, someone must be able to trust the result. Manual verification labor is being automated quickly. Tests, CI, staged rollout, and reversible deployment all reduce line-by-line human review. The assurance function remains; what changes is who produces the evidence and how it is aggregated. Its limit is independence. The validation mechanism cannot fail in the same way as the behavior being validated. Tests written and passed by the agent itself do not constitute independent evidence.

Third, authorization and accountability remain. Most per-action confirmations will disappear because predefined scopes of authority serve the same purpose more efficiently. Authorization answers which actions are allowed, and a person or organization with the relevant authority must set that boundary. Accountability asks who chose to deploy, who configured permissions, and how responsibility is allocated after an incident. It may be distributed across the operator, engineering owner, manager, vendor, organization, and regulator. These arrangements must exist before runtime. Compressing them into “a human must be responsible somewhere in the loop” leaves the person who clicked approve as the default point of liability for implementation defects and deployment negligence.

For an agent with a clear objective that can run autonomously for a long time, none of these three responsibilities disappears. All move earlier: who set the objective and success criteria, where the permission boundary lies, who owns the system, who gets alerted when it drifts, and who has the authority to stop it. People touch the trajectory less often, yet each interaction carries more weight. Accountability becomes especially difficult because there is no recurring human click available as a convenient place to attach responsibility.

Following this direction forward, I expect the central bottleneck to shift as interaction declines. The question will move from whether people are willing to intervene to whether they can still understand the situation in time. The less a person is needed during normal operation, the less they know about the trajectory when an anomaly occurs. The situation that requires their judgment is also likely to be among the hardest. Bainbridge made this point about automated cockpits. Vaccaro’s meta-analysis approaches it from the other side: adding more human steps does not automatically pay off. An increasingly important capability for agent products will therefore be handing the right person enough information at the moment of interruption: relevant history, current state, evidence supporting the proposed action, remaining options, downstream consequences, reversibility, and required authority. A confirmation box with only Allow and Deny cannot do this, yet that is still what most products provide.

Benchmarks will continue telling us how difficult a problem a model can solve, and that curve remains important. Some evaluations are already broadening along adjacent dimensions. $\tau$-bench asks whether the same class of task can be delegated repeatedly. HiL-Bench asks whether an agent knows when it should seek help. SWE-Together records how many user corrections a trajectory requires. Their common difficulty is also the difficulty of this perspective: once the human is included in the system, the object being measured expands into the model and its surrounding delegation system.

People are not standing outside the agent waiting passively for a result. As agents improve at execution, control across long trajectories, and automated verification, people will intervene less in the execution itself. More human judgment will move upstream into how objectives are formed, what evidence is sufficient for adoption, and where authority boundaries should be drawn.