Methodology.
Real Design Benchmark tests whether AI can design new features for existing products. A model designs a feature from a written brief and the product's other screens, without seeing the real one. People then compare both designs without knowing which is which.
- Version
- 1.1 · Oct 14, 2026
- Tasks
- 116 from 52 products
- Models
- Claude Opus 5.5 (medium), GPT-6.1 Sol (medium)
- Judges
- Anonymous visitors
- Results
- Since Oct 12, 2026
Why real products.
Most design work adds a feature to a product that already exists, and the new screen has to fit the rest. Design has no objective answer, but every product has the screen its team shipped, so that is the comparison. “Real product” means the live screen, which may itself have been made with AI help.
What we measure.
- Preference.
- Which design is better for the task: A, B or About the same.
- Identification.
- Which design is AI, and how sure: Not sure, Pretty sure or Certain. 50% is chance.
Preference is asked first, before the player commits to which design is AI, and someone can spot the AI design and still prefer it. Judgments are made on static screens, so they don't cover how a feature works in use, its accessibility or its code.
The tasks.
Each task is one feature of a live web product. We capture the real screens ourselves at 1280 × 800, in our own accounts filled with fictional data.
- Detail views.
- A list item opened into its full view. 56 tasks.
- Dialogs.
- A modal, drawer or menu opened over a page. 43 tasks.
- Related pages.
- A new page reached from an existing one. 15 tasks.
- Empty states.
- A page before it holds any data. 2 tasks.
We drop a task when the real screen shows something the AI couldn't reproduce from its inputs, such as a photo it was never given or a browser tooltip.
What the AI receives.
- Feature brief.
- Who it's for, what they need to do, and its rules and states. No layout, components or wording.
- Product screens.
- The screen the feature opens from and up to three others from the same product.
- Product kit.
- Fonts, icons and color tokens from those screens.
- Held out.
- The real screen, and any icon or style that appears only on it.
The model decides the layout and writes its own copy.
How a pair is made.
- 01Choose context.Other screens are checked against the real one for shared text, matching regions and near-duplicates, then reviewed by a person.
- 02Write the brief.An agent that never sees the real screen writes the brief from its text. A linter rejects layout and component words, and a second agent that can see the screen rewrites anything that still gives the design away. A person then checks every brief for leaks and mistakes.
- 03Generate.Each model runs in its maker's agent (Claude Code for Claude, Codex for GPT) in a clean workspace with file and shell tools only: no MCP servers, memory or skills. The model writes one HTML page, renders it, compares it with the product screens and revises.
- 04Audit.Every tool call is logged. A run is discarded if it leaves its workspace or uses the network.
- 05Review & export.A person checks each pair against the curation rules. Both sides are rendered at 1.5×, then scaled down and encoded the same way, so sharpness and file size give nothing away.
How a round works.
- 01Judge.Players see the feature's name and goal and two screens, A and B, in random order. They pick the better design, then the AI one and how sure they are.
- 02Reveal.The answer names the AI screen, its model and the product. Answers can change until then.
- 03Say why.Optionally, what gave it away and whether they use the product.
Players are anonymous: no accounts, a random id per browser and an optional self-reported role. The answer stays on the server until a player submits, and both images come from neutral addresses with the same size and headers, so nothing on the page gives it away.
How results are counted.
Results are shares of rounds, not of people. Only a player's first answer to each pair counts.
- AI identification rate.
- Rounds where the AI design was identified. 50% is chance.
- AI design preference rate.
- Rounds where the AI design was preferred. Real product and About the same share the same total.
- By model.
- Identification and preference for each model, on the pairs every model has answers for.
- Breakdowns.
- The identification rate by confidence and by product familiarity, and how often each cue was named.
Pairs stay private while they're in play, since publishing them would give the answers away.
See all resultsLimitations.
- Players choose to take part and roles are self-reported, so they aren't a representative sample.
- Frequent players count more.
- People who use a product may recognize its real screen. Familiarity is asked so those rounds can be separated.
- The products are self-serve web apps, mostly business software.
- Each model is tested in one setup, its maker's agent at medium effort. Results describe those setups, not AI design in general.
- Not every pair has every model yet, so models are compared only on the pairs they share.
- The model may have seen these products in training.
- Our accounts hold sparse fictional data, while the model writes its own sample content. That may favor the AI on preference.
- No confidence intervals yet. Small groups, such as a role with few rounds, can swing widely.
About.
Real Design Benchmark is sponsored by Subframe. Any change to this procedure will be dated here.
Sponsored by Subframe