What Makes Generated UI Good?
How we know a generated interface is actually good
TL;DR
Model launches often back the claim “good at UI” with a few polished screenshots or a sped-up recording. The result may well look good, but nothing shows whether it came from one generation or was picked from many, or which specific requirements it actually meets. A UI is good on particular requirements, and each one counts as evidence only after a person or a program has checked it against a clear standard. Take two scenarios: a 3D house built from listing photos, which a renter walks through once or twice, and a lab booking page used for months, with new requirements every week or two. Requirements differ first in how long they must be observed: overflowing text shows in a single frame, whether booking works shows only after actually booking a slot, and whether a new training rule breaks old bookings shows only weeks later, when the requirements change. They differ second in what they are checked against. Light leaking through gaps in a wall or text overflowing its box are defects in any artifact, recognizable without knowing what the artifact should be (integrity). Of the design polish in how an artifact presents itself and responds to use (craft), only a small part can be written as rules. Which wall the bedroom window belongs on can be settled only against the listing photos (correctness). And if the renter’s parents only swipe on a phone and never use arrow keys, even an accurate house is of no use to them; that standard (fit) is held by particular users and becomes known only when real users arrive with a purpose or are asked. Whether a check can be automated depends on whether its standard exists before generation and who holds it.
Today’s automated evaluation mostly takes two forms: a vision-language model (VLM) scores screenshots against a rubric, or a browser agent follows predefined steps and reports a pass rate. Both apply rules that benchmark designers wrote before generation. A VLM separates artifacts of very different quality and produces stable model rankings, but it judges poorly when two versions are close, and often misses geometric defects a few pixels wide, such as text overflow. An agent agrees closely with people when it runs tests that people wrote or reviewed, and misses most defects when it decides for itself what to test. Four kinds of requirements are rarely evaluated: animation without a reference recording, the difficulties first-time users and users with disabilities actually face, uses beyond the prompt, and changes over weeks. Training relies on the same cheap, automatically computed signals at scale, which suggests that capabilities improve in a rough order. Moving from broken to whole improves fastest, and a tidy first impression has likely improved along with it. The correctness of features and content comes next, since tests need expected results written in advance and are costly to run in a browser, and checking content needs a reference. Once interfaces are generally tidy, candidates differ only in details that VLMs cannot rank reliably, so higher-level polish slows down, and the four rarely evaluated kinds of requirements lag furthest behind. Rising leaderboard scores therefore say little about these deeper requirements. Part of the missing checking can happen at delivery, where the system serves a single request and the user who holds the standard is present. It can ask about the intended use before changing anything, turn the user’s expectations into tests rerun after every change, and keep each version so a broken change can be rolled back. The difficulties that first-time users and users with disabilities face, however, still surface only when those people use the artifact themselves.
An earlier post asked why generated interfaces so often look good at first glance, and what we are actually judging when we say a model has “taste.” “Good at UI” has since become a near-standard claim in model launches, almost always accompanied by a few polished screenshots or a sped-up screen recording. A prompt goes in, and moments later the screen shows a refined landing page, a playable racing game, or a 3D house you can walk through. These recordings usually cut out the middle and keep only the opening prompt and the final result. In between, the model may have spent tens of minutes on its own, writing code, taking screenshots to check its work, and revising again, and the result on display may be the best of several generations.
Consider two scenarios. The first is a 3D house. A renter has found an apartment they like and gives a model the dozen or so photos from the listing page, hoping to get a house they can walk through in the browser. Tens of minutes later, the screen does show a living room, a kitchen, and two bedrooms, with furniture in place and sunlight coming through the windows, and the keyboard moves you from room to room. The second is a lab booking page for shared instruments. After an agent builds the first version, lab members book and cancel slots on it every day, and the records are kept in a shared table. Every week or two a new requirement comes up: add a new instrument, let only trained members book it, or switch the calendar to a monthly view.
The renter will look at the 3D house once or twice and then close it, so what matters is whether the floor plan is accurate and matches the photos. The booking page will be used for months, so what matters is that booking and cancellation never go wrong, and above all that no change to the requirements wipes out the existing data. A static screenshot freezes one moment from one viewpoint. It can show whether the living room layout is clear, but checking the other rooms against the photos means switching through several viewpoints and comparing. Whether booking actually works can only be learned by clicking through it. Whether a revision broke an old feature becomes clear only after several more rounds of changes, or when the next version goes live.
Spending tens of minutes of machine time generating code is a compute cost, and that cost keeps falling as models and hardware improve. Simulating clicks and checking submitted results can also be handed to a program, provided someone has written the test cases and expected assertions in advance. Even checking whether the rooms match the photos can be handed to a multimodal model. The one thing compute cannot speed up is the next requirement. What new problems lab members will run into, and what new restrictions they will want, can only be learned by actually waiting several weeks.
People who know the field will point out that UI already has a full set of automated evaluation methods. A model that can read images (a VLM) receives a screenshot, acts as a judge, and scores it against an evaluation rubric; this is the familiar LLM-as-a-judge setup. For more involved cases, an agent that can operate a browser (a browser agent) follows predefined steps, and the task pass rate is reported. Many public UI benchmarks build on one of these methods or combine them, and they can evaluate an artifact in seconds to minutes. What this pipeline can tell us depends entirely on what the VLM can distinguish in a screenshot, which actions the agent actually performs, and what the predefined rubric and test cases contain. When benchmark designers write those rules, they have none of the actual listing photos the renter is working from, and they cannot anticipate the lab’s new requirements a few weeks later.
What counts as UI, and what each kind must get right
Interfaces generated in a chat window today go well beyond web pages and apps. They include slides, posters, charts, internal tools, games, physics simulations, 3D scenes, and even CAD models. Claude’s Artifacts let users revise web pages, SVGs, flowcharts, and interactive React components repeatedly within the same window (Anthropic, 2026), and Gemini’s Canvas can generate and preview web apps, small games, and interactive simulations directly (Google, 2025). These outputs once belonged to different trades, yet they are generated in much the same way. The model first writes code or a structured description, such as HTML, SVG, a component tree, or a 3D scene definition, and a browser, renderer, or other runtime turns it into something people can see and operate. Throughout this article, artifact means any such model-generated output that can be rendered or run. A static picture of a web page drawn directly by a model counts at most as a visual proposal. It has no runnable interactions and no underlying code to check or modify, so it falls outside this discussion.
Every artifact, whatever its kind, has two sides: content and presentation. Content is what the artifact expresses: whether its data, formulas, and text are accurate, and whether the entities it builds match reality. The content of the 3D house is the apartment itself, namely how many rooms there are, where the doors and windows sit, and how the living room connects to the kitchen, all of which must match the listing photos exactly. Presentation concerns how that content reaches the viewer. Information hierarchy, composition, typography, and color determine what people see first. In the 3D house, the spatial layout should become clear before the grain of the wooden floor does.
By their main mode of use, artifacts fall roughly into four kinds (Figure 1 shows how they spread out). The boundaries between kinds are not sharp, and no kind ranks above another. The 3D house, for instance, has to convey spatial relationships the way a floor plan does while also supporting a first-person walkthrough, and an accurate, information-dense data chart may well be harder to build than a simple input form.
Beyond the shared requirements of content and presentation, each kind of artifact carries one core requirement of its own. A presentational artifact is for reading and browsing. In slides, posters, flowcharts, and charts, the visual arrangement has to serve the logic of the content strictly: if a bar chart’s vertical axis does not start at zero, a small fluctuation in the data looks like a doubling. A task-oriented interface, such as a form, a dashboard, or an internal tool, supports operations, and its core requirement is that state carries through correctly from one step to the next. The booking page is a typical case. Choosing a slot, submitting, and cancelling all read and write the same shared records, and when two people try to book the same slot at once, the system must turn the later request away and must never silently overwrite the earlier booking. In an interactive experience, such as a physics simulation, an interactive explainer, or a game, the interaction itself is the core of the content, so input, state, and time have to stay tightly coordinated. Wherever a slider is dragged, the quantity bound to it must follow in real time, and pressing reset must cleanly return the system to a consistent initial state. A spatial artifact, such as a 3D scene or a CAD model, also has to handle the projection of a three-dimensional structure onto a two-dimensional viewport. When the 3D house is viewed from another angle, occlusion and perspective have to stay consistent, and the camera must never clip into a wall when turning a corner or passing through a door. When one dimension of a CAD part changes, the geometry of the linked assembly has to update with it.
The 3D house and the booking page also differ sharply in lifespan. The 3D house is often just a novelty to look at, while each version of the booking page is the foundation for the next. Adding an instrument, adding a training requirement, and switching to a monthly view are all changes made to a page that already holds months of real data. For an artifact that keeps evolving, every iteration has to meet the new requirement while protecting what already exists. Past bookings must never be lost, the existing cancellation logic must still work after the training requirement is added, and even wording that an administrator tweaked by hand in the source code should not be silently erased the next time the model regenerates everything. None of these risks leaves a trace in a screenshot of the first version; they depend entirely on how well the parts of the code are decoupled. If the booking data and the calendar view are independent, switching to a monthly view only requires replacing the display layer, and the stored bookings stay untouched.
How much of this the model has to handle on its own depends heavily on the environment it generates in. In a standalone canvas such as Claude Artifacts or Gemini Canvas, the model has to write complete code from scratch and manage content, visuals, and interaction by itself. Under a declarative protocol such as Google’s A2UI, the model only specifies which components to call and what data to bind, and the host application draws them from a set of pre-approved components (Google A2UI Team et al., 2026). The host then covers most of the visual presentation and basic interaction, and whether a component breaks the rules is also easier to check automatically and statically. When the model generates within a team’s existing design system (shared components, styles, and conventions) and a mature codebase, as with Claude Design (Anthropic, 2026), the visual and interaction conventions are already in place, and the model’s core task becomes keeping new code consistent with the existing design assets.
Even if a model can build a 3D house from photos that never clips through walls, gets the space right, and is ready to deliver after a single generation, that does not show it can build a good lab booking page. The booking page has to chain several steps into a reliable interaction flow and survive repeated changes over months. Talking about being “good at UI” without naming a setting tends to hide the real difficulties. The phrase rests on at least three unstated conditions: what kind of artifact is being built, whether it is used once or maintained over time, and whether the model writes code from scratch in a sandbox or generates within an environment whose existing conventions catch some of its mistakes.
How long to observe, and what to check against
Launch demos usually follow a carefully planned happy path. The camera glides from the entrance of the 3D house into the living room, and the booking page appears only in its empty initial weekly view. Viewers have no way to judge the corners the camera never reached or the states that were never triggered. Judging whether a requirement is met means sorting the evidence along two dimensions according to its nature (Figure 2). The first is the observation horizon: a single static frame, a sequence of continuous operation, or versions spanning several rounds of changes or even several weeks. The second is the standard the judgment is made against: a floor of quality every artifact has to hold (integrity), the design polish in how the artifact presents itself and responds to use (craft), the objective facts that the content and runtime results have to match (correctness), and whether the artifact serves a particular user’s actual purpose (fit).
The first screen you see on opening an artifact is only one of countless possible frames. New frames appear only when you walk into another bedroom, enter a very long title, narrow the browser viewport, or load real business data, and many broken surfaces, occlusions, and text overflows often first show up at that point. A sequence can only be checked by operating the artifact for a few minutes with a concrete task in mind. Whether the camera catches on the door frame when passing through a doorway, or whether a submission actually writes a record to the underlying table, can only be learned by working through it. Differences across versions can only be observed once new requirements arrive. After the training requirement is added to the booking page, one has to check both that the new rule works and that the historical records and old features are intact. The longer the observation horizon, the more someone has to interact in depth or wait patiently. Watching a recording cannot substitute for this. The people who can compare versions are usually those who actually take over maintaining the artifact: maintainers, collaborators, and the model itself when it tries to fix the code in the next turn of the conversation.
Light leaking through gaps in 3D walls, objects clipping through one another or sitting out of alignment, and text overflowing its container are unambiguous defects whatever the floor plan or the text. Recognizing them does not require knowing what the artifact is supposed to be, and this general floor of quality is what integrity means. Whether the lighting in a room looks flat, or whether the camera sits at a natural eye level, depends instead on aesthetic experience accumulated in trades such as photography and interior design. Craft refers to this kind of design refinement in how an artifact presents itself and how it responds to operation. It usually measures how well something is done and rarely has a clear boundary between broken and working. The parts of craft that can be distilled into fixed rules, such as keeping lines of body text under seventy or eighty characters, can be checked automatically just like integrity.
By contrast, which wall the bedroom window belongs on can only be judged against real photos of this apartment, and which record should appear in the underlying table after a booking is submitted can only be verified if someone has written down the expected result in advance. Judgments like these depend on a specific objective reference outside the artifact. The corresponding standard is correctness, which covers both the accuracy of factual content and the implementation of functionality. The extra core requirement that each kind of artifact carries, such as state transitions in the booking system or the mapping from slider position to parameter response in a physics engine, mostly falls under correctness, and it can often be observed only through a stretch of continuous interaction.
Another kind of evaluation can reverse completely when the user or the setting changes, even with an identical external reference. Suppose the renter wants to share the 3D house with their parents far away to get their opinion, and the parents only ever preview things by swiping in a mobile browser and would never use the arrow keys to move through a space. Then the 3D house, however accurately reconstructed and beautifully rendered, is of no use to them. Evaluating this requires knowing who the artifact is for and what problem it is meant to solve, which is the question fit answers. Whether a first-time user can get started smoothly is usually called usability in design, and usability spans two standards. General design guidelines, such as giving immediate feedback after an action and making a mistaken tap easy to undo, hold for everyone and belong to the craft of interaction. Where a particular user gets stuck depends on their background, habits, and cognitive preferences, and that belongs to fit.
The four standards rest on different grounds, and they also become available for automated checking at very different times. A check for broken surfaces and clipping only has to be written once to apply to almost every 3D scene. For visual atmosphere and lighting, only the few parameters already fixed in design guidelines can be checked in advance, and the vast majority has to be judged on the spot by experienced people looking at the finished artifact. Whether the windows are in the right place can be checked only once the specific listing photos are available. Whether a 3D walkthrough actually helps can be known only once a real user arrives with a definite purpose, or by asking them directly during the interaction.
A 3D house with intact walls and excellent lighting can still have its bedroom window on the wrong wall, and neither an automated script that checks for broken surfaces nor a designer attentive to lighting can catch it. In real generated web apps, scores on different evaluation dimensions are likewise only weakly correlated. WebCraftBench has models evaluate an app’s appearance, usability, and requirement fulfillment separately. The appearance evaluation is based mainly on static screenshots and roughly corresponds to craft. Requirement fulfillment checks the acceptance items listed in the requirements one by one, and more than 80% of those items concern functionality and content, which corresponds to correctness. Across individual artifacts, the Pearson correlation between these two scores was only 0.32, where 1 would mean a perfect positive linear relationship (Liu et al., 2026).
How UI is evaluated today
Long before models took on interface generation, design and engineering had built evaluation traditions spanning decades. Visual taste, functional correctness, and usability each had mature methods for checking it. Most of today’s automated evaluation, whether VLMs scoring screenshots or agents running scripted operations, has a prototype among these classic methods.
Existing methods in design and engineering
To judge whether an interface looks good and is well crafted, design teams usually rely on critique. A designer presents the work, and colleagues go through it screen by screen, examining visual hierarchy, alignment, spacing, and color, pointing out layouts that fail to guide the eye, and suggesting changes. This kind of evaluation draws on the aesthetic intuition of experienced practitioners and involves no quantitative scoring. Critique works because the team has first agreed on the design goals; its central task is to analyze whether the design achieves those goals and to frame feedback against them (Gibbons, 2016). Without that shared understanding, critique tends to collapse into individuals stating their own preferences.
To check whether features work and whether the interface is broken, engineering relies mainly on test programs. An end-to-end test fixes a sequence of operations and expected results. On the booking page, it might select a slot, click submit, and then assert that a new record has appeared in the underlying table. A visual regression test compares a newly rendered screenshot pixel by pixel with a baseline image that a person has approved. It checks whether the interface’s appearance has drifted, and the standard of good and bad was settled when that baseline was first approved. End-to-end UI tests are themselves highly prone to intermittent failures (flakiness), so the test pyramid in software engineering has long recommended a broad base of unit tests and very few end-to-end tests. The intuitive experience and usability of an interface have always been hard for test scripts to assess, and they still depend on people doing exploratory testing (Vocke, 2018).
The cheapest way to check usability is heuristic evaluation. A few evaluators go through the interface screen by screen against about ten general design guidelines, such as whether the system gives immediate feedback on actions and whether there is a clear way to undo a mistake. On the booking page, this means asking whether a “Booked” message appears after submission and whether a mistaken cancellation can be undone with one click. Nielsen describes these guidelines as broad rules of thumb, and how they apply to a specific interface depends entirely on the evaluator’s understanding at the time. Different evaluators tend to find very different problems. Across six real projects, Nielsen found that a single evaluator caught only about 35% of the usability problems on average, so the usual practice is to have three to five experts review independently and then pool their findings (Nielsen, 1994).
User testing invites target users to try the interface directly, to see how other people misunderstand an interface its designer knows well. A test usually recruits a few representative users to perform typical tasks while thinking aloud, and an observer records where they hesitate or get stuck and which controls they misread. Nielsen estimated that five users from the same group reveal about 85% of usability problems, and therefore recommended small, quick iterations: test only five people per round, fix what they reveal, and then run the next round (Nielsen, 2000). Once a system is deployed, online A/B experiments lasting weeks or even months can compare old and new versions using real users’ behavior. Changes their designers were confident about often turn out disappointing on real data. Summarizing online experiments at Microsoft, Kohavi reported that of the changes teams designed to improve a metric, only about a third produced a significant positive effect, and most of the rest made no difference or even hurt the metric (Kohavi, 2015). Google’s experiments on search ads likewise found that users may take months to adapt to interface changes, so short-term metrics do not necessarily predict long-term effects (Hohnhold et al., 2015).
Set against these traditional methods, having a VLM score static screenshots against a rubric amounts to letting a single evaluator stand in for both design critique and heuristic evaluation. This evaluator combs the screens for faults against a fixed rubric written by someone else and has never agreed on design goals with the product team. Sampling several times or convening a panel of models is extremely cheap and can smooth out some of the random variation in scores, but a panel of human experts works for a different reason: different people have different blind spots. If mainstream models share a perceptual limitation for some type of visual or geometric defect, adding more models will still miss it. Having an agent operate a browser according to a predefined script likewise amounts to running a limited set of end-to-end tests, and it only ever covers the paths written into the script. What user testing and online experiments capture is precisely the misunderstandings designers did not anticipate and the long-term changes in behavior, and both require real people to put in real time. In public UI benchmarks, the ordinary users who have the actual need rarely try an artifact and then judge it, apart from arenas where crowdsourced users vote for a rough comparison. People mostly appear in writing the tasks, reviewing the tests, and checking the model scores.
Five common approaches
By how they observe and what they judge against, existing UI benchmarks fall roughly into five approaches: reference reconstruction, screenshot scoring, scripted testing, multi-turn editing, and blind arena voting. VLM scoring and browser agents are the core tools used across them.
| Approach | How it observes | What it judges against | What it checks |
|---|---|---|---|
| Reference reconstruction (Si et al., 2025) | Renders in a window of the same size and compares text, position, and color with the original, block by block | The reference image, compared by a program | Whether the rendering matches the reference image |
| Screenshot scoring (Zhang et al., 2025) | A script takes screenshots before, during, and after operation; the VLM usually also reads the code | Rubric items written in advance for each task, checked one by one by a VLM | Whether the rendering is broken or refined, and whether the rubric items are met |
| Scripted testing (Lu et al., 2025) | A script or browser agent follows written steps | Written expected results | Whether the written steps can be completed and the results match expectations |
| Multi-turn editing (Li et al., 2026) | Edits over several rounds by instruction within one session | The target page for each round, compared by a VLM and by pixels | Whether the region to change was changed correctly and everything else stayed the same |
| Arena voting (Vichare, 2025) | Voters preview and operate two apps | The voter’s own preference | Which one the voter prefers |
In terms of observation horizon, reference reconstruction and screenshot scoring mainly examine frames; even multi-state screenshots are only a few discrete slices. Scripted testing examines the sequence of a single interaction, and multi-turn editing examines how code evolves across rounds within one session. Apart from the arena, all four approaches have their standards fixed by the benchmark designers before generation: the target reference image, rubric items written for each task, test scripts and assertions, and each round’s editing prompt and expected interface. Whether the judge is an automated comparison script or a VLM, it strictly applies these predefined rules. A VLM occasionally generalizes and catches problems outside the rubric, but the judgments that can be reproduced with confidence remain limited to the items written explicitly into the specification.
Arenas give up predefined standards. In WebDev Arena and its successor Code Arena, a real user typically states an open-ended request, apps generated by two models appear side by side for the user to try, and the user casts a blind vote for the winner. Of the five approaches, only the arena lets the people with the actual need make the judgment themselves, so implicit preferences never written into the prompt, such as the cognitive habits of a particular audience or the setting of use, can enter the evaluation. Voters, however, do not necessarily use these apps for real work. Their trials usually last a few minutes of poking around, and they rarely check whether the underlying data is correct. Impressions of appearance, interaction feedback, and fit to the request blend together, and the vote cannot reveal which dimensions the user actually weighed. Such casual votes often diverge noticeably from professional review. WebDevJudge had experts re-judge deployed WebDev Arena samples blind. With ties allowed, the experts agreed with the original crowd votes only 53% of the time, and even among samples with a clear winner, agreement was only 77.4% (Li et al., 2025).
Asking a VLM to score
In a VLM-based evaluation pipeline, the judge model usually receives both the page screenshot and the source code, and some benchmarks add several screenshots from before and after interaction along with operation logs. Many experiments comparing judges with human evaluation show that their agreement depends mainly on two variables: how large the quality gap between the candidates is, and whether that gap is clearly visible in the inputs.
In WebCraftBench, the model’s pairwise comparisons agreed with human judgment 85.3% of the time overall, and more than 90% of the time when the two artifacts differed widely in score (Liu et al., 2026). Yet when Luera et al. studied how well VLMs predict crowdsourced users’ subjective impressions, they found that once two interfaces looked comparable in quality to human participants, the model’s preferences became little better than random guessing (Luera et al., 2025). WebDevJudge draws its test set directly from WebDev Arena, where the competing apps mostly work. The best judge (GPT-4.1) agreed with expert labels about 70% of the time, against a human baseline of about 85% reported in the paper (Li et al., 2025).
Iterative revision of software, however, takes place precisely between similar versions, where an evaluator has to pick the better of two subtly different versions and also say exactly what to change next. In 2024, Duan et al. fed GPT-4 the structured metadata of Figma mockups and asked it to suggest changes based on design principles. Designers checked the suggestions one by one. In the first round, 52% were accurate. After designers made a first set of fixes following the suggestions, the overall accuracy of later rounds fell to 39%, which the authors attributed to the designs themselves having improved through revision (Duan et al., 2024). As design quality rises, the remaining defects become smaller and more scattered. The falling accuracy of the model’s suggestions and the declining agreement of judges on similar, high-quality pairs probably reflect the same kind of bottleneck.
The WebDevJudge team also examined systematically what role a rubric actually plays. When two human experts judged freely without a rubric, their agreement including ties was only 65%, with most of the disagreement over whether a pair should count as a tie. With a carefully written human rubric, expert agreement on a different set of samples reached 92%. The effect did not carry over to models. Whether a model compared strictly against the rubric or judged directly, its agreement with experts showed no clear difference, and the rubric helped only slightly when single pages were scored independently (Li et al., 2025). Human experts each have their own aesthetic scale, and a rubric’s value lies in aligning the team’s understanding and giving everyone a shared anchor, the same logic as calibrating goals before a design critique. A single model, however, already carries fixed internal priors, and a rubric written as text cannot get past its real blind spots in perceiving screenshots.
A good part of the judges’ agreement with experts comes from reading code. In WebDevJudge, GPT-4.1 agreed with experts about 61% of the time when it saw only screenshots and about 70% when it saw only code, and giving it both screenshots and source code performed about the same as code alone (Li et al., 2025). ArtifactsBench captures three screenshots, before, during, and after operation. Gemini-2.5-Pro’s agreement with the rankings of front-end engineers was about 75% from screenshots alone and about 79% from code alone, and combining the code with the three screenshots raised agreement to about 91% (Zhang et al., 2025). The experts look at the actual running page, yet judges reach 70 to 80 percent agreement just by reading code. This is likely because, most of the time, what the code says is what the page shows. When the code is sound but the rendering has occlusion or a distorted layout, the code reveals nothing, and such defects can be exposed only by looking at the rendering.
Even when the judge does rely on vision, input images are heavily downsampled before they reach the vision encoder, and small pixel-level flaws are often erased in the process. According to Claude’s technical documentation, newer models cap the long side of an image at 2576 pixels, and earlier models at only 1568 pixels (Anthropic, 2026). In a typical 1440×6000 full-page screenshot, standard 14-pixel body text is left with an effective height of only about 6 pixels and 3.7 pixels, respectively, after resampling. Capturing only the first viewport to preserve fidelity simply drops the interactive areas below it from the evidence.
Less obvious is that models still often fail at basic spatial relationships even when the image detail is clear enough. BlindTest’s experiments on mainstream models from 2024 showed that on simple geometric judgments such as counting how many times two lines cross, four mainstream models averaged only about 58% accuracy. Yet a lightweight linear classifier trained directly on the vision encoder’s intermediate hidden states could judge whether circles overlap or lines intersect with more than 99% accuracy (Rahmanzadehgervi et al., 2024). This suggests that, at least for these spatial and geometric relationships, the vision encoder has already captured the information, and the loss more likely happens when visual features are translated into judgments in natural language. In real UIs, common defects such as overlapping or misaligned components and text overflowing its container are relative geometric relationships of the same kind, and large-scale tests in UI-Lens bear this out. On nearly five thousand Chinese-language interfaces labeled by designers, nine mainstream general-purpose VLMs averaged an F1 score of only about 22% at detecting text overflow (F1 balances precision and recall, with a maximum of 100%) (Xiang et al., 2026). Targeted fine-tuning may help with this weakness. In a study focused on multi-window split screens on Android, giving a fine-tuned model both the screenshot and the control coordinates raised its F1 for detecting unintended control occlusion to 87.2% (Zhang et al., 2026), although both its defect definitions and its interfaces differ from general web settings.
A static screenshot captures only one rendering in a particular browser at a particular size, which is why test frameworks such as Playwright require regression screenshots to be taken in a consistent environment (Microsoft, 2026). Evaluating across browsers, devices, and resolutions, Guo et al. found that 68% of generated web pages had compatibility problems in at least one environment, most often a broken layout. Feeding the model screenshots from every environment does not necessarily help, though. When GPT-5.5 diagnosed cross-platform compatibility problems, its F1 was 85.1% with only the DOM tree as input and dropped to 77.9% once screenshots were added (Guo et al., 2026). The authors believe the multiple screenshots added too much visual noise and distracted the model.
Letting an agent operate
When people write the steps and expected assertions of the test cases, or review them line by line, browser agents execute them with high reliability and catch dynamic defects that static images cannot show. In WebGen-Bench, GPT-4o drafts the test cases and people then review each one. An agent follows these scripts to drive the browser and judge the results. Its verdicts on individual tests agreed with manual testing 86% to 94% of the time, and the average end-to-end pass rate it computed across websites almost exactly matched the manual figure (26.4% versus 26.0%, for example) (Lu et al., 2025).
Things change once the browser agent tests on its own. WebTestBench gives the agent only the requirements and leaves it to come up with test points and diagnose failures. None of the models tested exceeded an F1 of 30% at detecting defects, and most had recall below 25%. The team attributes this to default-correctness bias: unless the model sees clear evidence of an error, it tends to assume the page works (Kong et al., 2026). The test points the models listed themselves covered less than 70% of the human checklist, and most of what they missed were things the requirements did not state explicitly. Giving the agent the human-written test points improved F1 (Claude Sonnet 4.5, for example, rose from 21.9% to 49.2%), but F1 still stayed below half. The failures were spread across every stage: failing to trigger an interaction, misreading the page state, or getting the assertion wrong after obtaining the output.
At the critical step of judging whether an interaction produced the expected result, evaluation runs into the classic test oracle problem of software engineering: the system needs a mechanism independent of the code under test to decide what the correct behavior is (Barr et al., 2015). When people write tests, they write down the expected results. When an agent judges on the fly or an LLM generates assertions automatically, the model has to supply the oracle itself. Konstantinou et al. showed that models generating assertions from existing code tend to match the code’s actual behavior, locking bugs already latent in the code into the expected values (Konstantinou et al., 2024). This tendency to accept whatever is observed as correct behavior probably has the same root as assuming a page works whenever no clear evidence of an error is visible.
A harder problem is that some errors live only in internal state and leave no trace on the surface of the interface. WebGrader documents a representative case. After the user clicks an add button, the interface shows a success message as expected, yet no data is actually added to the underlying table. WebGrader marks this as a failure by comparing the stored state before and after the interaction (Chen et al., 2026), whereas anyone looking only at the success message would conclude that the operation succeeded. In interactive 3D scenes, judgments based on internal state and purely external observation barely agree. WorldCoder requires every generated interactive scene to follow a specification that continuously syncs runtime state into an internal object, and checking logic written by experts then reads that object automatically to assert correctness. The team systematically compared three purely external methods (a DOM parser, multimodal VLM screenshot scoring, and a browser agent allowed up to 8 interaction steps) with this internal-state check. The rankings from all three external methods showed almost no statistical correlation with the internal-state assertions. Measured by the rank correlation Kendall $\tau_b$ (1 means perfect positive correlation, 0 means no relationship), even the best group, the agent, reached only +0.111 (Lu et al., 2026).
Part of the reason for such a large gap is that the two kinds of evaluation rest on different premises. Among the cases where WorldCoder’s internal-state check gave a score of zero, 42.8% occurred because the code did not expose the state object as the specification required, so the checker could not read any data and marked it wrong, even though the rendering itself was not necessarily broken. More telling are another 40.8% of cases, in which the rendering looked reasonable and self-consistent, but the backend data never advanced after the user triggered an interaction. Such problems are invisible in static screenshots. Even an agent operating the page can easily be fooled by a normal-looking screen if it relies only on visual feedback; only a direct check of the internal data state reveals them.
Beyond errors hidden in internal state, interaction tests are themselves prone to intermittent failures. Romano et al. analyzed 235 flaky test cases from traditional UI testing and found that 45.1% came from asynchronous timing races: the test script asserted before the page had finished updating or a network request had settled, so the test reported an error although the page was fine (Romano et al., 2021). The same problem remains when an agent does the operating, and some of the false alarms in WebTestBench came from delays in asynchronous rendering. A coarse task pass rate alone cannot tell whether the app under test has a logic bug or the agent missed a click or asserted too early. Finding out requires seeing which steps the tester took and what it saw at each one.
From one artifact to a model score
What a benchmark finally presents to the public is usually a model score condensed into a single number. What that number can mean depends on several steps in turning individual artifacts from single runs into a score: how candidates are sampled, how sub-metrics are combined into one score, how scores across tasks are aggregated, and whether the artifacts a model produces are highly uniform. A given result from a model may be a single generation, the best of 10 sampled candidates picked by a person, or the outcome of several rounds of automated debugging and patching by an agent behind the scenes, and the final static output makes these hard to tell apart. A single generation reflects what an end user will most likely experience in everyday use. The best result picked afterward by the author measures the ceiling the model occasionally reaches in the long tail of its outputs. If the selection logic is itself built into the product, as in a design workflow that generates four layouts at once for the user to choose from, it measures the combined performance of the system, model generation plus reranking. The final product of several rounds of debugging measures what the model produces together with a complex autonomous workflow; the 3D walkthrough the renter waited tens of minutes for belongs here. If a report hides the selection and iteration process, leaderboard rankings are systematically inflated. A quantitative analysis of Chatbot Arena found that vendors who privately test multiple model variants and release only the best one systematically raise their relative position on the leaderboard (Singh et al., 2025). A launch demo that records the best of several generations works the same way.
Once several metrics have been measured for an artifact, a common practice is to average them into a composite score. Equal weighting of this kind often flattens important structural conflicts. A 3D model with stunning rendering whose floor plan is oriented differently from the photos, and another whose floor plan is exact but whose lighting is flat, can end up with the same average. For a user deciding whether to rent the apartment, only the second is clearly of real value. In addition, when metrics for different dimensions actually draw on the same source of information, they end up counting it more than once without anyone intending it. Visual similarity, DOM layout similarity, and VLM screenshot scores look like three independent algorithms, but all three rest on a snapshot of the static rendering. Adding them together quietly triples the weight of static visuals in the overall score.
In large benchmarks, averaging over many samples does smooth out occasional noise in individual evaluations. In ArtifactsBench, the model ranking aggregated from VLM scores closely matched the overall ranking built from real users’ crowdsourced blind votes on WebDev Arena, with 94.4% agreement between the two rankings, where 100% would mean identical rankings (Zhang et al., 2025). This high agreement shows only that the VLM matches the average of public preference across the whole distribution. Whether it comes close to professional design judgment in head-to-head comparisons still depends on the accuracy of its individual pairwise decisions. WebGen-Bench had GPT-4o, o3, Claude, and human judges each score page appearance. The absolute scores shifted from judge to judge, but the relative order of the generation systems barely moved (Lu et al., 2025). This consistency shows only that these judges agree with one another, and if they share a systematic blind spot, a stable ranking may well hide a shared bias. Automatic judges are therefore far more reliable for ranking models in aggregate than for deciding which of two individual samples is better. Averaging stops helping altogether in two important situations. The first is a widespread, systematic perceptual blind spot. If a VLM cannot see text overflow, a model that frequently produces overflow never receives the penalty it deserves on the leaderboard, and however stable the leaderboard is, it cannot show how well that defect is controlled. The second is a setting that requires single decisions, such as online iteration. During multi-turn editing, the system has to choose at every step between two versions that differ only very slightly, and each wrong choice leaves behind worse intermediate code.
Whether they score artifacts independently or compare them in pairs, all five existing approaches end up averaging scores computed artifact by artifact, and they rarely examine the diversity of what a model generates. As a result, a model that can only apply one particular template fluently has a chance to score high on every isolated task. The common patterns analyzed in the earlier post, such as a white background with a subtle purple gradient set in the Inter typeface, are hard to detect under current artifact-by-artifact evaluation.
In short, VLMs do well at telling apart artifacts of very different quality, and the overall model rankings they produce, once averaged over many samples, are stable enough. Faced with two similar versions that both work, pixel-level layout flaws, or data breaks hidden inside the runtime state, however, they misjudge easily, and no text rubric, however detailed, can make up for their perceptual blind spots at the level of the raw visual input. Likewise, a browser agent that strictly follows high-quality test cases prepared by human experts executes them about as accurately as professional manual regression testing. Once it is allowed to explore on its own, deriving its own test points and deciding whether results are correct, it misses large numbers of hidden defects because it has no oracle grounded in the real requirements.
Which requirements are still rarely evaluated
Current UI benchmarks very rarely evaluate at least four key kinds of requirements in any systematic way, and when they touch them at all, they stay at the edges that an automated script can execute: open-ended animation without a reference recording, the usability difficulties that first-time users and users with disabilities actually face, intended uses beyond the prompt that were never written down, and code changes that accumulate over weeks. Animation has to be examined as something that unfolds over time, and version changes have to be compared across several rounds of history. The barriers a real audience faces and the intentions it leaves unstated have to be tied closely to specific people and the actual tasks they bring, and that is the core of the fit standard.
Animation
When a user cancels a booking and the corresponding row vanishes instantly, it is hard to tell at once which row was removed. A short, natural collapse animation makes clear which time slot is being freed. This kind of temporal continuity lives in the transitions between adjacent frames, yet current automated evaluation gives the model at most a few scattered slices of the whole process. The three screenshots before, during, and after an operation that ArtifactsBench relies on are a product of exactly this discrete sampling, and the paper’s authors explicitly acknowledge that such static slices can hardly capture the real smoothness of complex interactions (Zhang et al., 2025). Sampling more densely requires screen recording. The browser automation framework Playwright records video at 25 fps by default (Feldman, 2026), but the video is usually downsampled again before it reaches a large model. Gemini, which understands video natively, samples one frame per second by default (Google, 2026), and WebVR, a benchmark that asks models to rebuild a page from a recording, also feeds the model only a few dozen uniformly spaced frames, or samples at two frames per second (Dai et al., 2026). In a real interface, a typical 200 ms transition occupies only 5 frames of a native 25 fps recording. At one frame per second, the whole micro-animation is very likely to fall between two sampling points, and the model never sees a single frame of it (Figure 3).
enlarged
Even when the key frames are captured, models often struggle to distinguish subtle temporal dynamics. TimeBlind gave multimodal models pairs of contrasting short videos in which the subject, background, and geometric relationships are identical, and the only variable is how an action unfolds over time; in one video a person walks progressively faster, for example, and in the other progressively slower. Against a random-guessing baseline of about 6%, the best of the more than twenty frontier models tested reached only about 48% accuracy, while human participants reached 98% (Li et al., 2026). These tests on ordinary physical actions show that models still have clear deficiencies in general temporal perception, such as telling apart nonlinear changes over time.
When a definitive reference recording exists, animation evaluation has found partial solutions. WebVR and Animation2Code are both reconstruction tasks: the model receives a video recording of interactions with the target page and has to write front-end code that reproduces exactly the same dynamic effects, and the temporal features of the generated recording are then compared with those of the reference. WebVR has a VLM review the generated recording together with the underlying code, item by item against a detailed rubric, with interaction and animation forming a separate dimension. On the overall quality of whole sites, the best judge agrees with UI/UX experts 96% of the time when choosing between two candidates, but the paper does not report agreement for the animation dimension alone. With only the recording as input, the evaluation struggles to separate the subtle differences among the strongest models, and with only the code as input, even a blank page that renders nothing visible can score well (Dai et al., 2026). Animation2Code tries to isolate temporal consistency numerically. It tracks feature points in screen space and measures how far the generated animation deviates from the original recording in motion vectors and instantaneous speed. When several automated program metrics and VLM scores were each used to predict which candidate animation crowdsourced annotators preferred, on each of the three dimensions examined (appearance, temporal dynamics, and overall quality) at least one purely programmatic metric came closer to human choices than the VLM did (Ji et al., 2026).
When a new interface is generated purely from a natural-language request, there is no ground-truth video to align frame by frame, and whether the animation’s timing works can usually be decided only by a model judging without any reference. No automatic animation judge has yet been validated. Moreover, “appropriate animation timing” has no standard answer. Material Design guidelines suggest about 300 ms for transitions on mobile and 150 to 200 ms on desktop (Google, 2016). The same animation may feel sluggish on a large desktop screen and just right on a small touchscreen. A fair judgment of an animation requires first establishing the device it runs on and the state change it is meant to convey.
Usability and accessibility for real users
Even when every booking feature works, a first-time user can still run into obstacles. The control for cancelling a booking may have no text label, or the page may give no feedback after submission, leaving the user unsure whether the booking went through. A test script follows preset steps, while a newcomer has to figure things out alone. In existing benchmarks, usability is mostly scored by models. WebCraftBench computes its usability score by having a model operate the application itself and record the process and each state, and then having an evaluation model score the record. GPT-5.6-Sol, which ranks first on appearance, ranks sixth on usability (Liu et al., 2026). How closely this score tracks real users’ experience has not yet been checked against people. Only its rubric would show which part of usability it actually covers, and it probably covers only the craft half of usability. Luera et al. studied a simpler case, in which a VLM predicts crowdsourced users’ subjective impressions from a single screenshot. When judging which of two interfaces is easier to use, the model agreed with the crowd’s choice only about half the time, roughly as often as a coin flip would (Luera et al., 2025).
Having a large model check screenshots screen by screen against classic usability guidelines takes over the form of heuristic evaluation directly, with an automated system in the evaluator’s seat. Zhong et al. gave GPT-4 the 3 to 9 screenshots left behind as users performed tasks in two mobile apps, asked it to flag violations of usability guidelines, and cross-checked its findings against a manual review by five experienced experts. Of the pooled problems found by everyone, GPT-4 covered 73% for the rental app, while the five experts together covered 57%. A closer breakdown tells a different story. Among the continuity problems that only become visible across a sequence of pages, the model caught 3 of 7, while the experts together caught 6. More seriously, 43 of the problems the model raised turned out on review to be unfounded false alarms, against only 6 false reports from the experts (Zhong et al., 2025). The model searches broadly, but it easily misses interaction breaks that span pages, and its list mixes in many false alarms that someone still has to sift through one by one.
Heuristic evaluation, after all, only checks pages against guidelines and cannot replace real users trying the interface, so some researchers have tried to simulate user interaction with multimodal models. First-click testing measures where users click first. Kuric et al. collected the first clicks of more than three thousand real participants across 45 tasks and compared them with clicks simulated by GPT-4.1. In 53% of the tasks, the model’s click positions differed significantly from those of real people. Adding personas, chain-of-thought, or adjusted sampling parameters did not narrow the gap; these only made the simulated results look more credible (Kuric et al., 2026). The authors of UXAgent also state explicitly that simulated agents are suited to debugging a study design before formal user research and cannot replace testing with real people (Lu et al., 2025).
Accessibility evaluation asks whether people with disabilities can use an interface. If the booking page shows whether a slot is full only through color, people with color vision deficiencies cannot tell, and screen readers cannot convey what the color means. Accessible design has the mature WCAG guidelines and automated checking tools, but the tools are good at checking whether static attributes are missing and whether color contrast is sufficient. Some criteria can only be checked by changing system settings. WCAG 2.3.3, for example, requires that motion animation triggered by interaction can be turned off, unless the animation is essential (World Wide Web Consortium, 2026). Checking it requires the test environment to simulate the “reduce motion” setting (prefers-reduced-motion) and verify that the animation stops.
Automated tools alone also cannot determine whether a page is accessible enough, and W3C’s evaluation guidance stresses that manual review is indispensable (World Wide Web Consortium, 2026). The UK Government Digital Service (GDS) built a test page with 143 planted accessibility failures. Ten checking tools together found only 71% of them, and 42 failures were missed by every tool, including the common problem of distinguishing links by color alone (Duran, 2017). Tools can check whether a label exists, but they cannot judge whether it conveys the content appropriately. Calò et al. point to a “semantic accessibility gap”: if an image’s alternative text is written as alt="image", it passes syntax checks, and a screen reader reads the meaningless word “image” to a visually impaired user. They used a model to scan 300 interfaces generated by three frontier models and reported 541 semantic problems that passed rule checks but carried no real meaning, more than half of them ambiguous buttons and links. These problems were reported by the model and were not verified one by one by people (Calò et al., 2026).
Beyond guidelines and tools, real audiences ultimately have to take part. W3C explicitly encourages involving people with disabilities and older adults in testing, to uncover real usability problems that compliance audits alone cannot reach (World Wide Web Consortium, 2026). Suppose every slot in the booking system has a correct ARIA label and a screen reader reads each one out without error. A user who relies on speech output, facing seven days a week with a dozen or more consecutive slots each day, may still have to sit through dozens of announcements to find one open slot. This heavy interaction fatigue never shows up in any list of code lint rules. In published research, work that systematically tests AI-generated interfaces with users with disabilities is still rare.
What users want it for
The prompt “build a house from these listing photos” specifies only the form of the output, not what the user wants to do with it. A renter who wants to measure whether their sofa will fit in the living room might write exactly this prompt, and so might a renter who wants to send a walkthrough link to their parents to look over on a phone, yet the two pull the implementation in very different directions. Generating to a generic average standard may happen to suit some people, but for those whose need it misses, the artifact is useless for the actual problem, however complete it looks. The trouble is that when a benchmark checks items only against the original prompt, a version that guessed right and a version that missed entirely receive exactly the same score. Explicit constraints that can be read directly off the prompt are easy to verify once they are turned into rubric items or automated test scripts (this is the part that “requirement fulfillment” and automated assertions in existing benchmarks check). Many implicit constraints, however, are never written into the prompt, and even the renter often only remembers that they need to measure the sofa after seeing a rough first version.
To reach what lies outside the prompt, existing benchmarks can only try to work out by hand, in advance, what a hypothetical user leaves unsaid. InteractWeb-Bench sets up a simulated user agent holding a set of hidden constraints that were never written into the prompt and are released only when the model actively asks, to test whether the generating model can recognize missing information and clarify the requirements (Wang et al., 2026). In its experiments, all nine mainstream models scored above 3.9 out of 5 on grasping the gist of the explicit requirements, yet their rate of drawing out the hidden constraints by asking was generally below 40%. Most of the time, the models treated the incomplete instructions as a definitive specification and implemented them in full. X+Slides constructs three audience personas with different needs for the same source material, has a large model weight information elements according to each audience’s background, and then evaluates whether the generated slides address that group’s core concerns (Chen et al., 2026). What such evaluations capture, however, are only personas preset in the lab. Which audience the particular person in front of the screen belongs to, and in what situation they use the artifact, remain unknown to the benchmark.
Changes over weeks
Existing evaluations of multi-turn code editing almost all take place within a single session. MT-Web2Code has the model keep editing from its own previous round of code. Compared with starting each round from the correct previous version, Kimi-K2.6 scored 23.8 points lower overall (out of 100) by the fifth round (Li et al., 2026). FronTalk similarly found that models easily break previously working features in later rounds (Wu et al., 2026). EvoGenUI-Bench evaluates five consecutive rounds of edits. The best model averaged a 74.9% pass rate per round, but the share of sessions with all five rounds correct dropped to 37.3% (Peng et al., 2026). Looking only at the pass rate of a single round makes it easy to overlook the problems that accumulate over successive edits.
More importantly, the sequence of edits in these benchmarks is usually a script written in advance by the benchmark authors. In a real software lifecycle, the training requirement for the lab booking system often comes up only a few weeks later. By then the system already holds a large amount of real historical data, interspersed with description text the administrator has edited directly, and benchmark authors can hardly imagine these states on the first day. Static code analysis can give theoretical estimates of maintainability, but for UI there is still no empirical evidence on how well such metrics predict whether the next real change will succeed. Assessing realistically how well an artifact withstands such changes requires following it as it evolves over weeks or even months, recording each change request, each diff of the source code, and the historical assets quietly lost after each change. Public datasets that record real requirements and changes to UI artifacts over several weeks remain rare.
Signals used in training
Models that generate interfaces output code, but the qualities that decide whether the experience is good can only be perceived once the code is rendered and run: how the layout unfolds, how animations connect, and whether the state transition after a click is exact. Pretraining on vast amounts of code and documentation gives the model the basic skills of syntax and implementation. How the code looks and feels once rendered, however, can only be learned through a runtime environment that actually runs the code and turns what it observes into quantitative feedback. This feedback works along much the same lines as evaluation: either a VLM scores screenshots, or an agent runs test scripts. In an academic evaluation, running this process on a few hundred test samples is enough to finish the study. In a reinforcement learning (RL) training loop, the system has to keep scoring an endless stream of sampled candidates. Inviting real people to use an artifact in depth for weeks is feasible in an occasional experiment, but putting it directly into a high-frequency training loop is impractical to engineer. What a training pipeline can actually consume are mostly machine signals that need no human involvement and can be computed automatically for each sample in a very short time.
Signals from rendered frames are plentiful and cheap
In existing public training work, synthesizing data, cleaning demonstration data, and even designing reward functions mostly rely on static rendered frames. The cheapest data to obtain are pairs of code and web page screenshots. WebSight’s large corpus was synthesized entirely by an automated pipeline: one language model came up with website requirements, another wrote the HTML, and the pages were then rendered and screenshotted automatically in a headless browser, cheaply producing two million cross-modal pairs (Laurençon et al., 2024). In supervised fine-tuning (SFT), the filtering of demonstration data is likewise anchored closely to static renderings. WebCode2M collected human ratings for ten thousand real web pages, trained a model to score page quality, and used it to filter out more than half of a huge pool of raw crawled pages as low quality (Gui et al., 2025). UICoder has the model iteratively generate interface code itself and keeps, as data for the next round of training, only samples that compile successfully and that the cross-modal representation model CLIP judges to match the text description closely (Wu et al., 2024).
When WebGen-R1 designs rewards for reinforcement learning, the main scores, past a basic threshold, also center on the rendered page. The code has to pass syntax checks and render successfully. A VLM then gives appearance and functionality scores from 0 to 5 on screenshots of several routes, and console errors are added as a basic functionality score. The browser agent is called only in offline evaluation and is not part of the RL reward (Jiang et al., 2026). Across these practices, most signals concern whether the code compiles, whether the page looks good, and whether images and text match, and the interface is rarely operated during training.
Preference alignment training compares candidates using people or a reward model. UIClip starts from real web pages and algorithmically degrades their text contrast, alignment, and spacing, creating more than two million pairs in which the original is better (Wu et al., 2024). These synthetic negatives differ sharply from the originals, so training on them amounts to teaching a model to recognize clearly broken pages. Choosing between two designs with no obvious flaws, by contrast, often requires a designer’s judgment. Wu et al. asked 21 designers to rank model-generated interfaces, write comments, draw sketches, or edit the designs directly, collecting 1,460 pieces of feedback (Wu et al., 2026). Each of the four kinds of feedback was converted into pairwise comparisons and used to train its own reward model. All of these designer reward models continued training from UIClip, with less than a thousandth as much new feedback as UIClip had synthetic samples, and each was then used to train a generator. In human evaluation, the generator trained with sketches clearly beat the one trained with UIClip as its reward model. The generator fine-tuned on direct edits had a slight advantage that did not reach significance, and the generators trained on rankings and comments even fell behind UIClip. When six researchers reviewed a sample of these annotations, preference judgments derived from rankings alone agreed with the researchers only 49.2% of the time, while samples from direct edits reached 76.1%. One possible reason is that ranking compresses a designer’s complex tradeoffs into a single label that others find hard to reproduce consistently, whereas an edited page puts the specific change directly into the layout.
Relying on these signals also tends to make designs look alike. Web design itself has been converging: a study tracking the form of websites from 2003 to 2019 found that their layouts began to converge from around 2007 (Goree et al., 2021). In analyzing why models favor the Inter typeface and purple gradients and rarely add animation on their own, Anthropic notes that such safe choices, which are hard to get wrong, dominate the pretraining data (Rajasekaran et al., 2025). An automatic scoring model may likewise learn the safe choices that most annotators rate highly. Using it to filter data or to drive reinforcement learning can raise the average, but it may also push every artifact toward the same few looks.
Signals from running and editing are still scarce
Running multi-step operations in a real browser before giving a reward costs far more, and little work actually uses it in training. WebGen-Agent tries to bring interaction evaluation into reinforcement learning. Each step’s reward is a weighted combination of a screenshot appearance score and the accuracy of tests executed by a browser agent. With it, Qwen2.5-Coder-7B’s functional accuracy rose from a baseline of 12.4% to 38.9% after supervised fine-tuning, and to 45.4% after reinforcement learning (Lu et al., 2025). WebGrader has a model write operation flows and expected results from the requirements, drives the page through them with test scripts while recording the rendered page and the internal data state, and then has a verdict model judge whether each flow meets its goal. Flow pass rate makes up 70% of its reward and appearance 30%. Starting from the same base model, the model trained with WebGrader reached 52.01% functional accuracy on the benchmark, above the 42.66% of the group trained with a VLM plus a browser agent. Used directly as a judge on test pages with injected faults, a browser agent recalled only about 60% of the faults (Chen et al., 2026).
Lighter alternatives settle for less and check only whether the page responds at all after an interaction. Qwen3-Coder-Next automatically enumerates interactive nodes from the DOM tree and simulates clicks on them, then has a VLM compare the screenshots taken just before and after each click, filtering out samples whose interactions are broken and produce no response (Qwen Team, 2026). Whether the response matches the business rules, however, still needs rigorous expected assertions. MiMo-V2-Flash feeds full interaction videos recorded with Playwright to a video model for scoring, and the team believes this misreads the rendered page less often than looking at static screenshots (Xiaomi LLM-Core Team, 2026). Although video preserves the full evidence of animation, the technical report gives no experimental data on whether the model can actually recognize unnatural stutters and distorted transitions.
Data for training code editing is harder to build, because each example needs an initial version, an editing request, and a correctly edited target version. Most existing data stays within a single session. VisRefiner starts from seven thousand pages, perturbs some of them with rules, collects the model’s own mistakes on others, and rewards edits by how far they advance the fix (Deng et al., 2026). Holding multi-turn conversations does not necessarily teach a model how to edit code, however. On a chart-code task, MM-ReCoder found that when the model edited freely for two rounds and was rewarded on the second round, nearly half of the samples simply copied the first round’s code unchanged in the second round, and only 3.4% were real fixes. When each group of samples shared the same first round and training focused specifically on how to edit, the share of improvements rose to 14.4%, along with an 8.4% risk of making things worse (Tang et al., 2026). Training data like the lab booking page, which spans months and requires adding new features while protecting existing data and hand-customized code, does not yet exist at any scale in public sources.
What the reward does not see
Models readily exploit loopholes in reward functions. UICoder used CLIP to evaluate whether a screenshot matched the requirements and found that CLIP sometimes gave a high score whenever the page contained text with the prompt’s keywords, forcing the team to add a similarity constraint against reference images (Wu et al., 2024). When GLM-5 was trained on slide layout, its rewards included geometric rules and a whitespace ratio. The model learned to truncate and drop long text that did not fit, or to adjust spacing excessively to produce a layout that passed (GLM-5 Team, 2026). Text no longer overflowed, but content was lost, because no reward checked whether the content was complete; the team responded by improving the renderer. Rewards that compare versions before and after an edit show similar behavior. ReLook penalizes edits that make things worse, and the model responded by changing as little as possible (Li et al., 2025). When MM-ReCoder tried rewarding the second round’s improvement over the first, the model learned to submit deliberately worse code in the first round so that it could score in the second (Tang et al., 2026).
In accessibility optimization, A11yn weights axe-core violations by severity, normalizes them by DOM size, and uses the result as a penalty. After reinforcement learning, the model’s violation rate under axe-core fell by 87.5%, but checked with AChecker, which played no part in training, it fell by only 58.1% (Yoon & others, 2025). The gap of nearly 30 percentage points between the two tools may mean that some of what the model learned targets axe-core’s specific rules, or it may simply reflect differences in the rules the two tools cover. More importantly, rule checks of this kind all examine attributes, and a purely rule-based reward still cannot catch placeholders like alt="image" that are formally compliant but semantically meaningless.
The order of progress
Each reinforcement learning update aggregates rewards over a batch of samples. Occasional misjudgments mainly make the signal noisier and slow training down. What cannot be averaged away are two kinds of systematic bias. One is the blind spots the reward model cannot see, which never produce negative feedback at any point in training (Figure 4). The other is whatever the reward systematically overrates, which the model exploits again and again during generation to collect points, as happened with UICoder and GLM-5. With methods like GRPO, which compute rewards from the relative standing of candidates within a group, signal quality likely declines as training goes on. Early on, candidates in a group differ widely, with some failing to render and others producing broken layouts, and a VLM can easily tell them apart. Once the model has the basic form down and candidates differ only in fine details, the VLM’s scoring error on similar samples swamps the actual differences, the relative wins and losses mix in more random guessing, and further improvement slows.
Following this logic, one can infer that the first breakthrough usually comes in moving from broken to whole: pages render properly, and layouts no longer fall apart on a large scale. Both belong to integrity and can be checked without knowing what the artifact is supposed to be. Whether a page renders is decided by a program, and large-scale layout collapse is easy for a lightweight script or a VLM to catch, so these signals are cheap and plentiful. Early in training, candidates differ enormously at this level, which steadily provides clear comparisons to learn from. The work in the literature that reports before-and-after training data also concentrates almost entirely on such programmatic checks. WebGen-R1 makes rendering a hard requirement. With supervised fine-tuning alone, only 30.69% of its outputs rendered validly, rising to 95.89% once reinforcement learning was added (Jiang et al., 2026). ReLook’s end-to-end training doubled the valid rendering rate from about 40% to about 80% (Li et al., 2025). The share of GLM-5’s slides that fit a 16:9 aspect ratio rose from 40% to 92%, though the report does not say how much of that came from truncating content (GLM-5 Team, 2026). Published research rarely gives before-and-after comparisons of whether layout breakage declined at the same time, so this step rests more on inference. Integrity also contains blind spots that these signals hardly reach. A VLM often cannot see slight text overflow or controls overlapping by a few pixels, and even coordinate rules hard-coded from the DOM box model to block them may be satisfied mechanically, as GLM-5 did by truncating content.
The visual impression at first glance belongs to craft. Across the range where differences are obvious, this ability has likely improved along with integrity: the phenomenon described at the opening of the earlier post is exactly that frontier models can already produce neatly laid out, evenly spaced interfaces in a single generation. Where this progress comes from is hard to disentangle with the available material. Safe designs in the pretraining corpus, curated fine-tuning demonstrations, and appearance rewards from VLMs during reinforcement learning may all play a part. The multimodal model used to score appearance before and after training is also often of the same kind as the model providing the reward, so a rise in those scores cannot be taken directly as evidence of real aesthetic progress.
Correctness comes next, covering both whether features work and whether content is correct. Interactive features show themselves only under dynamic operation, and if the expected results can be written down in advance, they can be turned into automated tests. A scripted run in a real environment can check stored state that screenshots cannot see, and it gives far more reliable feedback than looking at the rendered page; WebGrader used exactly this kind of signal to train a model with more stable interaction logic. The costs of this approach are just as clear. The expected results have to be written correctly in advance, and having a model write them leads straight into the oracle problem. Running multiple steps in a browser also costs far more than taking a single screenshot, so such signals are limited in number during training and cover only the few test paths written in advance.
Whether content is correct can be handed to a program for comparison whenever a clear reference exists. When a page is reconstructed from a design mockup, a script can check text, coordinates, and color one by one, and such tasks can be synthesized at scale, as WebSight did. When CAD parameters are generated from three orthographic engineering views, the code can be run to compute how much the resulting 3D solid overlaps the target, and Ortho2CAD uses this volumetric overlap as a reinforcement reward (Joglekar et al., 2026). Deterministic algorithms perform these comparisons, so they are unaffected by what a VLM cannot see, and they are cheap, which means progress on tasks with a definite reference may be fast. Most real UI requirements, however, are a sentence of natural language with no ready-made design to check against piece by piece. In these open-ended settings, checking content can only fall back on VLM judgment or on tests, so for correctness as a whole, cheap and trustworthy reinforcement signals still cover only a limited part. The apartment photos the renter supplies are also a factual reference, but photos and a 3D scene in the browser are hard to match one to one, so the comparison ultimately has to be handed to a multimodal model looking at images. How reliable such a cross-modal check is has not yet been measured publicly.
Once models clear the basic threshold and produce interfaces without obvious flaws, telling which composition better suits particular content, or which set of details is more refined, becomes a completely different matter. Candidates usually differ only subtly in these details, mainstream VLMs cannot give a stable preference in close comparisons, and progress in craft at the level of finer polish slows down. Model-generated designs therefore tend to converge on similar safe states that show no obvious flaws at first glance and closely resemble one another; the common generated styles discussed in the earlier post reflect exactly this tendency. Revision sketches from professional designers can give concrete direction, but human annotation is expensive and hard to supply continuously at the scale training requires.
Slowest to progress are the four areas that are rarely evaluated systematically: animation without a reference recording, the difficulties that first-time users and users with disabilities actually face, uses beyond the prompt, and changes over weeks. Some automated substitutes exist for these, but they have either not been carefully calibrated against real experience or have been found to deviate substantially when compared, as with the simulated agent whose click positions differed from real people’s in more than half of the tasks. When such uncalibrated tools are used as rewards, it is hard to say whether the model has learned the requirement itself or has merely learned to please the checking tool’s own biases. As for uses beyond the prompt, current public training data and reward designs do not even show any signal aimed at them yet.
If a requirement can be written as a repeatable program, and that program’s judgments have been carefully compared with those of professionals, training gains a new signal that is both cheap and reliable, and progress on that dimension may come earlier. If the occlusion detector trained for Android multi-window split screens transferred to general web pages, layout defects a few pixels in size might also decline earlier. When a ground-truth recording exists, the temporal feature metrics used by Animation2Code can cheaply check animation timing. In accessibility, A11yn has already brought a rule checker into the reward computation, although the checker itself has not yet been compared against human judgment.
This inference carries two uncertainties. First, progress in functional logic does not necessarily depend entirely on dedicated UI rewards. The skills in debugging code and reasoning about tests that models learn from general software engineering tasks are likely to transfer directly to front-end development, so functionality may mature faster than an inference from UI signals alone would predict. Second, most published research reports the before-and-after change for only one item, with some papers measuring the rate of successful rendering and others the task pass rate. No published experiment has yet rigorously compared, within a single joint training run, the rates at which these capabilities develop.
Checks that can be added at delivery
A generation system is the whole engineering environment wrapped around the model: the toolchain that launches the artifact, takes screenshots, and runs tests; the storage that keeps each historical version; and the orchestration that decides what to do next. Some checks are too expensive to run on every sample during training, and benchmarks rarely run them on each of hundreds of artifacts. At actual delivery, though, the system serves only the one request in front of it, so an expensive check is justifiable when it only has to run once. The system can return rendered screenshots, feedback from real operations, and earlier versions to the model, and it can also ask the user directly in the conversation. In a set of tasks that involve editing visualization code, with GPT-4o as the base model in both cases, direct single-turn generation passed only 55.75% of the tasks, while an agent that first plans, generates several candidates, renders and checks them, and then selects and refines the best one raised the pass rate to 67.99% (Rahman et al., 2026).
Letting the model see the page and the process
When WebGen-Agent generates a website, it returns rendered screenshots and the results of a browser agent’s operation tests to the model as two separate streams of feedback. Adding visual screenshots mainly improved appearance scores. Adding operation tests significantly improved functional accuracy but lowered appearance scores slightly, which the researchers explain by debugging functionality occasionally breaking an existing layout (Lu et al., 2025). New features can likewise easily break existing logic. TDDev has an agent run tests while developing a web application and compares several schedules for testing. One of them, a feature-by-feature mode, is designed specifically to prevent such regressions: each time a new feature is finished, every previously passing test case is rerun. The other two strategies test everything once the whole application is written, or leave it entirely to the agent to decide when to test (Wan et al., 2026).
How much external feedback helps depends on the tester’s own ability to diagnose problems. In TDDev’s experiments, with Claude Sonnet 4.6 as the base model and the agent deciding when to test, functional accuracy (measured by the benchmark’s human-written test set) rose from 67.8% without testing to 91.5%. When Qwen3.5 developed the application and tested it itself, however, no testing schedule brought a clear improvement. It often blamed its own mistaken operations in the browser on the application under test and judged correctly implemented features to be failures. When Claude Sonnet 4.6 ran the tests instead, the functional accuracy of the applications Qwen3.5 wrote rose by about 7 to 9 percentage points, although the improvement was statistically significant for only one testing schedule. Models reviewing their own rendered pages have a similar blind spot. An Anthropic engineering write-up notes that agents evaluating their own work tend to praise it heavily, and that a separate evaluator devoted to finding problems is often far more effective than asking the agent to reflect on its work. Subjective wording in a rubric, such as “museum quality,” can also push the generated results toward a particular stereotyped visual style (Rajasekaran, 2026).
The user is present and can be asked
At actual delivery, the user is right there at the interface, and the system can ask them directly what they plan to do with the artifact. When generating the 3D house, the renter may only want an overhead view to understand the floor plan, or may want to open the door and walk in. For the first purpose, a flat bird’s-eye view of the space is enough; for the second, the system has to build a smooth first-person walkthrough of the interior. While the requirement is still taking shape, generating a few clearly different candidates for the user to choose from can also help them work out what they actually want. The reasons users give also directly change which technical properties need to be checked next. If the renter has never mentioned placing furniture, whether the room dimensions are labeled accurately depends on what they want the house for. Once they say they want to know whether their sofa will fit in the living room, the system has to estimate the room dimensions and label them in the scene, and the accuracy of those estimates can in turn be checked against the doors and floor tiles in the photos.
When a system should ask questions, and how deeply, varies enormously across coding agents. SpecBench designs evaluation tasks in which an agent questions a simulated user and compiles a requirements specification, without writing any code. In one experiment, the agent had to infer, through a limited number of questions, the user’s leanings on 80 latent design decisions. When the number of allowed questions rose from 0 to 10, Claude Code and Cursor became more accurate in about 80% of task instances, whereas Gemini CLI improved in only 61% of them and, in 17% of the tests, submitted its result early without using all of its questions (Wang et al., 2026). In both SpecBench and InteractWeb-Bench, the respondents are virtual personas simulated by large models, and the SpecBench authors stress that simulated users differ from real people. The Human Is Part of the Agent discusses in more depth the central place people hold in judgments of this kind.
Keeping every version and editing onward
A system can compare versions, and roll back safely when things get worse, only if it records every version together with its check results. When a model repeatedly edits its own code, later versions are often no better than earlier ones. Xiong et al. tested how frontier models reconstruct pages from design mockups and then iterate on the code on their own. On the Design2Code test, which is built from screenshots of real web pages, GPT-5.4’s overall similarity (on a 100-point scale) rose slightly, by 0.76 points, after its first round of self-repair and then drifted downward with fluctuations, falling below the initial version in 10 of the 15 iterations. The researchers observed that while fixing one area, the model often broke another area that had been correct (Xiong et al., 2026). In this test, the best version happened to be the one right after the first round of edits, and a system that had not kept a complete history of snapshots would have no way to recover it. WebGen-Agent rolls back to the last stable state after consecutive runtime errors, and at the end selects the best version to deliver to the user based on combined operation-test and screenshot scores (Lu et al., 2025). When to stop iterating also depends on what can still be checked. Once all the preset hard rules pass, if the remaining differences concern only color, spacing, or particular business intent, automated processes often cannot tell better from worse, and human experience or the user’s own decision is needed.
In the lab booking page scenario, once the system has been running for three months, what needs maintaining is an interface that holds a lot of real data. The database stores several hundred past bookings, and the administrator has edited the description text directly. If a request then arrives to add a new lab instrument, the generation system has to implement the new feature while respecting the existing data and the traces of manual customization. If the description text lives in the code and the model simply regenerates from the old code in its context, the administrator’s edits may be silently overwritten. Judging by two representative tools, v0 and Claude Design, support for this is still very limited. In design mode, Vercel’s v0 first holds the user’s manual adjustments in a separate panel, and once the user applies them, it packages the intended changes into a prompt and has the model generate a new version (Vercel, 2026). Claude Design supports conversation, comments, and direct editing, but its official documentation notes that it does not yet support version history (Anthropic, 2026). Neither product’s documentation promises that the user’s manual adjustments will survive the next full revision. Checking whether old assets are intact also requires going down to the underlying data. After the layout changes from weekly to monthly, for instance, every booking moves to a different visual position, so comparing screenshots before and after cannot show whether any booking records were lost; only a direct comparison of the underlying data tables can settle it.
This delivery process also has costs that cannot be ignored. Part of the cost is machine compute, which can be measured: building a single application with TDDev consumes, on average, from just over one million to over seven million additional tokens (Wan et al., 2026). Another part shifts onto the user’s time. Even if the system makes its questions more specific and presents the differences between candidates more clearly, the user still has to spend effort sorting through them and deciding. Changes over weeks are even harder for automated tools to handle alone. Unless it has been recorded specifically, which old rules people still depend on is known only to those who have used the system over those weeks, and when something breaks, only they can notice it and repair it.
What it takes to call a UI good
A UI that is good is always good on a few specific requirements, and each requirement needs clear evidence behind it. When a model or tool claims to be “good at UI,” the first question is what kind of interface it is dealing with. In a task like the 3D house, what has to be guaranteed is that the scene renders, has no large-scale clipping or misalignment, and roughly matches the floor plan. These requirements can be checked the moment generation finishes, and existing screenshot scoring and scripts that check for broken surfaces do capture part of the quality. Once the question becomes whether the 3D model can be used to check furniture dimensions, or even to help decide whether to rent, comparing the floor plan against the photos still lacks any validated measurement. The lab booking page is a different situation. The logic that really matters is spread across operation sequences and version history, such as the records in the shared table when two people book at once, or whether old bookings survive the third change of requirements. For interfaces like this, leaderboard scores barely touch the core. More telling than a model’s benchmark score is whether the delivery system runs automated tests against expected results written by people, whether the model running those tests is itself reliable, whether the system asks about the intended use before changing code, and whether it can roll back safely when something breaks.
Inferring from current training and evaluation practice, the order in which capabilities improve depends mainly on how costly it is to obtain feedback signals. Moving from broken to whole improves fastest, and the accuracy of functional logic and factual content comes next. Once pages are generally tidy, progress in distinguishing finer craft slows down, and the four rarely evaluated areas lag furthest behind. These differences in pace are often invisible in technical metrics, because current benchmarks and model training rely on the same easily obtained surface evidence. A page that renders successfully with evenly spaced visuals earns steady rewards in training and full baseline points in benchmarks. The difficulties that first-time users and users with disabilities actually face, uses beyond the prompt, and problems arising in changes over weeks, on the other hand, are hard to penalize effectively in training and barely register on existing leaderboards. That is why rising leaderboard positions and increasingly polished output are in fact decoupled from whether those deeper requirements have actually improved.
Whether a check can be handed to an automated program depends mainly on whether the standard for judging it exists before generation, and on who holds that standard. General rules for code syntax and layout can be written once and reused. What the database should record after a booking is submitted, or what exactly the renter wants to do with the 3D model, can only come from people who understand the domain and from actual users. Once people write these requirements down at delivery as concrete expected results (for example, when two people book the same slot at the same time, the one who submits later should see that the slot is full), the system can rerun the check automatically every time it changes the code afterward. This provides a far more concrete constraint than scoring a few candidate versions or offering an abstract comment.
Looking ahead, the data needed for uses beyond the prompt and for changes over weeks may come naturally out of this delivery process. A user’s answers to clarifying questions, traces of manual tweaks on the interface, and regressions caught by automated tests together form samples that carry real context and the intent behind each edit. This inference applies only to uses beyond the prompt and to changes over weeks. Animation without a reference recording falls outside it, and the difficulties that first-time users and users with disabilities actually face cannot be filled in automatically from such delivery records alone either: the cognitive obstacles first-time users run into and the interaction fatigue users with disabilities experience surface only when those people use the artifact themselves. Turning this mechanism into reality also requires better tools: Claude Design does not yet support version history, and v0 does not promise that a full regeneration will leave code the user tweaked by hand intact. Whether training models directly on such delivery records brings real improvement has also not yet been tested in any published experiment.
References
- Anthropic. (2026). What Are Artifacts and How Do I Use Them? What Are Artifacts and How Do I Use Them? (Anthropic, 2026) paper
- Google. (2025). New Gemini Features: Canvas and Audio Overview. New Gemini Features: Canvas and Audio Overview (Google, 2025) paper
- Google A2UI Team, Salomon, I., & Yosef, L. (2026). A2UI + MCP Apps: Combining the Best of Declarative and Custom Agentic UIs. A2UI + MCP Apps: Combining the Best of Declarative and Custom Agentic UIs (Google, 2026) paper
- Anthropic. (2026). Introducing Claude Design by Anthropic Labs. Introducing Claude Design by Anthropic Labs (Anthropic, 2026) paper
- Liu, C., Zou, Z., Gao, P., Tao, J., & others. (2026). WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective. arXiv Preprint arXiv:2609.15387. WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective (Liu et al., 2026) paper
- Gibbons, S. (2016). Design Critiques: Encourage a Positive Culture to Improve Products. Design Critiques: Encourage a Positive Culture to Improve Products (Gibbons, 2016) paper
- Vocke, H. (2018). The Practical Test Pyramid. The Practical Test Pyramid (Vocke, 2018) paper
- Nielsen, J. (1994). The Theory Behind Heuristic Evaluations. The Theory Behind Heuristic Evaluations (Nielsen, 1994) paper
- Nielsen, J. (2000). Why You Only Need to Test with 5 Users. Why You Only Need to Test with 5 Users (Nielsen, 2000) paper
- Kohavi, R. (2015). Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years. Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years (Kohavi, 2015) paper
- Hohnhold, H., O’Brien, D., & Tang, D. (2015). Focusing on the Long-term: It’s Good for Users and Business. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Focusing on the Long-term: It’s Good for Users and Business (Hohnhold et al., 2015) paper
- Si, C., Zhang, Y., Li, R., Yang, Z., Liu, R., & Yang, D. (2025). Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering. Proceedings of NAACL-HLT, 3956–3974. Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (Si et al., 2025) paper
- Zhang, C., Li, Y., Xu, C., Liu, J., & others. (2025). ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation. arXiv Preprint arXiv:2507.04952. ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation (Zhang et al., 2025) paper
- Lu, Z., Yang, Y., Ren, H., Hou, H., Xiao, H., Wang, K., Shi, W., Zhou, A., Zhan, M., & Li, H. (2025). WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch. arXiv Preprint arXiv:2505.03733. WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch (Lu et al., 2025) paper
- Li, Q., Hu, S., Liu, H., Feng, X., Liu, S., & Wan, G. (2026). MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification. arXiv Preprint arXiv:2608.03474. MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification (Li et al., 2026) paper
- Vichare, A. (2025). The Next Stage of AI Coding Evaluation Is Here. The Next Stage of AI Coding Evaluation Is Here (Arena, 2025) paper
- Li, C., Zheng, Y., Huang, X., Fang, T., Xu, J., Chen, L., Song, Y., & Hu, H. (2025). WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality. arXiv Preprint arXiv:2510.18560. WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality (Li et al., 2025) paper
- Luera, R. A., Rossi, R., Dernoncourt, F., Basu, S., & others. (2025). MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces. MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces (Luera et al., 2025) paper
- Duan, P., Warner, J., Li, Y., & Hartmann, B. (2024). Generating Automatic Feedback on UI Mockups with Large Language Models. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642782 Generating Automatic Feedback on UI Mockups with Large Language Models (Duan et al., 2024) paper
- Anthropic. (2026). Vision. Vision (Anthropic, 2026) paper
- Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2024). Vision Language Models Are Blind: Failing to Translate Detailed Visual Features into Words. arXiv Preprint arXiv:2407.06581. Vision Language Models Are Blind: Failing to Translate Detailed Visual Features into Words (Rahmanzadehgervi et al., 2024) paper
- Xiang, W., Wu, Y., Chen, X., Li, X., & Chen, S. (2026). UI-Lens: Assessing General MLLMs’ Potential to Automate UI Display Quality Assurance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. UI-Lens: Assessing General MLLMs’ Potential to Automate UI Display Quality Assurance (Xiang et al., 2026) paper
- Zhang, X., Wang, R., Cui, J., Huang, H., Xue, W., Hu, W., Xiang, J., & Hao, R. (2026). Proactive Detection of GUI Defects in Multi-Window Scenarios via Multimodal Reasoning. arXiv Preprint arXiv:2604.19081. Proactive Detection of GUI Defects in Multi-Window Scenarios via Multimodal Reasoning (Zhang et al., 2026) paper
- Microsoft. (2026). Visual Comparisons. Visual Comparisons (Playwright, 2026) paper
- Guo, Z., Xiao, J., Sun, Y., & Huo, Y. (2026). Does It Render Everywhere? A Study of Cross-Environment Compatibility in MLLM-Generated Webpages. arXiv Preprint arXiv:2608.12518. Does It Render Everywhere? A Study of Cross-Environment Compatibility in MLLM-Generated Webpages (Guo et al., 2026) paper
- Kong, F., Zhang, J., Yue, Y., Sun, C., & others. (2026). WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing. arXiv Preprint arXiv:2603.25226. WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing (Kong et al., 2026) paper
- Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5), 507–525. The Oracle Problem in Software Testing: A Survey (Barr et al., 2015) paper
- Konstantinou, M., Degiovanni, R., & Papadakis, M. (2024). Do LLMs Generate Test Oracles that Capture the Actual or the Expected Program Behaviour? arXiv Preprint arXiv:2410.21136. Do LLMs Generate Test Oracles that Capture the Actual or the Expected Program Behaviour? (Konstantinou et al., 2024) paper
- Chen, B., Liu, H., & Zhang, S. (2026). WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader. arXiv Preprint arXiv:2608.06474. WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader (Chen et al., 2026) paper
- Lu, S., Xu, Y., Yu, K., Jiang, S., Yu, Y., Wang, Y., Yang, H., Zhang, Y., Wang, B., He, R., & Liang, J. (2026). WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis. arXiv Preprint arXiv:2606.01869. WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis (Lu et al., 2026) paper
- Romano, A., Song, Z., Grandhi, S., Yang, W., & Wang, W. (2021). An Empirical Analysis of UI-based Flaky Tests. Proceedings of the 43rd International Conference on Software Engineering. An Empirical Analysis of UI-based Flaky Tests (Romano et al., 2021) paper
- Singh, S., Nan, Y., Wang, A., D’Souza, D., & others. (2025). The Leaderboard Illusion. arXiv Preprint arXiv:2504.20879. The Leaderboard Illusion (Singh et al., 2025) paper
- Feldman, P. (2026). feat(video): allow recording video at a custom fps. feat(video): allow recording video at a custom fps (Playwright, 2026) paper
- Google. (2026). Video Understanding. Video Understanding (Google, 2026) paper
- Dai, Y., Lai, Y., Huang, M., Guo, H., & others. (2026). WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics. arXiv Preprint arXiv:2603.13391. WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics (Dai et al., 2026) paper
- Li, B., Zhao, K., Zhang, C., Mitra, C., Nyandwi, J. de D., & Bertasius, G. (2026). TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs. arXiv Preprint arXiv:2602.00288. TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs (Li et al., 2026) paper
- Ji, A., Mudunuri, A. V., Chan, D. M., & Suhr, A. (2026). Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation. arXiv Preprint arXiv:2606.28593. Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation (Ji et al., 2026) paper
- Google. (2016). Duration & Easing. Duration & Easing (Material Design, 2016) paper
- Zhong, R., McDonald, D. W., & Hsieh, G. (2025). Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation. arXiv Preprint arXiv:2507.02306. Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation (Zhong et al., 2025) paper
- Kuric, E., Demcak, P., & Krajcovic, M. (2026). What Would GPT Click: Practical Effects of Human-AI Behavioral Misalignment and the Cost of Synthetic Participants in User Experience. arXiv Preprint arXiv:2605.18302. What Would GPT Click: Practical Effects of Human-AI Behavioral Misalignment and the Cost of Synthetic Participants in User Experience (Kuric et al., 2026) paper
- Lu, Y., Yao, B., Gu, H., Huang, J., Wang, J., Li, Y., Gesi, J., He, Q., Li, T. J.-J., & Wang, D. (2025). UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706599.3719729 UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design (Lu et al., 2025) paper
- World Wide Web Consortium. (2026). Understanding Success Criterion 2.3.3: Animation from Interactions. Understanding Success Criterion 2.3.3: Animation from Interactions (W3C, 2026) paper
- World Wide Web Consortium. (2026). Evaluating Web Accessibility Overview. Evaluating Web Accessibility Overview (W3C, 2026) paper
- Duran, M. (2017). What We Found When We Tested Tools on the World’s Least-Accessible Webpage. What We Found When We Tested Tools on the World’s Least-Accessible Webpage (Duran, 2017) paper
- Calò, T., Gurita, A.-E., & De Russis, L. (2026). Measuring the Semantic Accessibility Gap in LLM-Generated Web UIs. Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3772363.3799364 Measuring the Semantic Accessibility Gap in LLM-Generated Web UIs (Calò et al., 2026) paper
- World Wide Web Consortium. (2026). Involving Users in Evaluating Web Accessibility. Involving Users in Evaluating Web Accessibility (W3C, 2026) paper
- Wang, Q., Hu, H., Chen, L., Wang, H., Alinejad-Rokny, H., Lin, Y., & Yang, M. (2026). InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation? arXiv Preprint arXiv:2604.27419. InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation? (Wang et al., 2026) paper
- Chen, H., Zhou, X., Zhou, W., Shao, X., Zhu, Y., Wang, B., Hong, J., Jia, A., & Wu, F. (2026). X+Slides: Benchmarking Audience-Conditioned Slide Generation. arXiv Preprint arXiv:2606.19256. X+Slides: Benchmarking Audience-Conditioned Slide Generation (Chen et al., 2026) paper
- Wu, X., Xue, Z., Yin, D., Zhou, S., Chang, K.-W., Peng, N., & Wen, Y. (2026). FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback. arXiv Preprint arXiv:2601.04203. FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback (Wu et al., 2026) paper
- Peng, Y., Xia, L., Wang, Z., Ye, J., Ning, K., & Wen, H. (2026). EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants. arXiv Preprint arXiv:2608.29387. EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants (Peng et al., 2026) paper
- Laurençon, H., Tronchon, L., & Sanh, V. (2024). Unlocking the Conversion of Web Screenshots into HTML Code with the WebSight Dataset. arXiv Preprint arXiv:2403.09029. Unlocking the Conversion of Web Screenshots into HTML Code with the WebSight Dataset (Laurençon et al., 2024) paper
- Gui, Y., Li, Z., Wan, Y., Shi, Y., Zhang, H., Su, Y., Chen, B., Chen, D., Wu, S., Zhou, X., Jiang, W., Jin, H., & Zhang, X. (2025). WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs. Proceedings of the ACM Web Conference. WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (Gui et al., 2025) paper
- Wu, J., Schoop, E., Leung, A., Barik, T., Bigham, J. P., & Nichols, J. (2024). UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback. Proceedings of NAACL. UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback (Wu et al., 2024) paper
- Jiang, J., Cai, C., Park, C., Shen, J., Kim, S., Li, J., & Wang, Y. (2026). WebGen-R1: Incentivizing Large Language Models to Generate Functional and Aesthetic Websites with Reinforcement Learning. arXiv Preprint arXiv:2604.20398. WebGen-R1: Incentivizing Large Language Models to Generate Functional and Aesthetic Websites with Reinforcement Learning (Jiang et al., 2026) paper
- Wu, J., Peng, Y.-H., Li, X. Y. A., Swearngin, A., Bigham, J. P., & Nichols, J. (2024). UIClip: A Data-driven Model for Assessing User Interface Design. Proceedings of the ACM Symposium on User Interface Software and Technology. UIClip: A Data-driven Model for Assessing User Interface Design (Wu et al., 2024) paper
- Wu, J., Swearngin, A., Vajjala, A. K., Leung, A., Nichols, J., & Barik, T. (2026). Improving User Interface Generation Models from Designer Feedback. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3772318.3791567 Improving User Interface Generation Models from Designer Feedback (Wu et al., 2026) paper
- Goree, S., Doosti, B., Crandall, D., & Su, N. M. (2021). Investigating the Homogenization of Web Design: A Mixed-Methods Approach. Proceedings of the CHI Conference on Human Factors in Computing Systems. Investigating the Homogenization of Web Design: A Mixed-Methods Approach (Goree et al., 2021) paper
- Rajasekaran, P., Wei, J., Bricken, A., Vorwerck, M., & Whitehead, R. (2025). Improving Frontend Design through Skills. Claude Blog. Improving Frontend Design through Skills (Anthropic, 2025) paper
- Lu, Z., Ren, H., Yang, Y., Wang, K., Zong, Z., Pan, J., Zhan, M., & Li, H. (2025). WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning. arXiv Preprint arXiv:2509.22644. WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning (Lu et al., 2025) paper
- Qwen Team. (2026). Qwen3-Coder-Next Technical Report. arXiv Preprint arXiv:2603.00729. Qwen3-Coder-Next Technical Report (Qwen Team, 2026) paper
- Xiaomi LLM-Core Team. (2026). MiMo-V2-Flash Technical Report. arXiv Preprint arXiv:2601.02780. MiMo-V2-Flash Technical Report (Xiaomi LLM-Core Team, 2026) paper
- Deng, J., Yao, K., & Zhang, L. (2026). VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation. arXiv Preprint arXiv:2602.05998. VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation (Deng et al., 2026) paper
- Tang, Z., Zhang, X., Yuan, J., Zou, Y., Gunjal, V., Jiang, S., & Modolo, D. (2026). MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22164–22173. MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correction (Tang et al., 2026) paper
- GLM-5 Team. (2026). GLM-5: from Vibe Coding to Agentic Engineering. arXiv Preprint arXiv:2602.15763. GLM-5: from Vibe Coding to Agentic Engineering (GLM-5 Team, 2026) paper
- Li, Y., Zhang, C., Lv, R., Liu, A., Deng, K., Zhang, Y., Liu, J., Zhou, W., & Zhou, B. (2025). ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Coding. arXiv Preprint arXiv:2510.11498. ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Coding (Li et al., 2025) paper
- Yoon, J., & others. (2025). A11yn: Aligning LLMs for Web Accessibility-Aware UI Generation. arXiv Preprint arXiv:2510.13914. A11yn: Aligning LLMs for Web Accessibility-Aware UI Generation (Yoon et al., 2025) paper
- Joglekar, A., Regmi, A., Shimada, K., & Kara, L. B. (2026). Ortho2CAD: 3D CAD Generation from Orthographic Drawings Using Vision Language Models. arXiv Preprint arXiv:2607.08891v2. Ortho2CAD: 3D CAD Generation from Orthographic Drawings Using Vision Language Models (Joglekar et al., 2026) paper
- Rahman, M., Azimlu, A., Rahman, S., Laskar, M. T. R., Bhuiyan, A., Joty, S., & Hoque, E. (2026). VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback? arXiv Preprint arXiv:2608.10408. VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback? (Rahman et al., 2026) paper
- Wan, Y., Liang, T., Xu, J., Xiao, J., Huo, Y., & Lyu, M. R. (2026). From Runnable to Shippable: Multi-Agent Test-Driven Development for Generating Full-Stack Web Applications from Requirements. arXiv Preprint arXiv:2605.17242v2. From Runnable to Shippable: Multi-Agent Test-Driven Development for Generating Full-Stack Web Applications from Requirements (Wan et al., 2026) paper
- Rajasekaran, P. (2026). Harness Design for Long-Running Application Development. Anthropic Engineering. Harness Design for Long-Running Application Development (Anthropic, 2026) paper
- Wang, H., Han, L., Xu, K., & Srivastava, A. (2026). Turning Intent into Specifications: A Benchmark and an Interactive User-Assistant Agent. arXiv Preprint arXiv:2606.20585. Turning Intent into Specifications: A Benchmark and an Interactive User-Assistant Agent (Wang et al., 2026) paper
- Xiong, T., Yang, Z., Wang, X., Lin, C.-C., Ma, R., Lin, K., Wang, Z., Li, L., Liu, C., Chen, R., Duraiswami, R., Huang, H., & Wang, L. (2026). Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation. arXiv Preprint arXiv:2608.24138. Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation (Xiong et al., 2026) paper
- Vercel. (2026). Design Mode. Design Mode (Vercel, 2026) paper
- Anthropic. (2026). Get Started with Claude Design. Get Started with Claude Design (Anthropic, 2026) paper