The cheap model needed the guide.
Four engines, seven conditions, twelve app specs. With the guide in context, the cheapest engine scored where the expensive ones did, at about a fifth of a cent per generated app.
vint is a small TypeScript UI framework we open-sourced. It is written to be used by a model rather than a person, and no model has been trained on it, so everything needed to use it ships in a guide that goes into context.
Real DOM, no virtual DOM, no JSX, no build step. Alongside the runtime it ships a guide of about three thousand words, a contract of thirty-nine numbered clauses, thirty-three error codes written as instructions rather than symptoms, and a test harness the model can run on its own output. The question was whether that is enough for a model to write a correct app on the first try and diagnose the failure on the second. The repository includes an evaluation that measures it; the other frameworks in the tables are there to give the numbers a scale, not because vint is competing with them.
How it was tested
Four engines — Claude Opus 5, Claude Sonnet 5, gpt-5.6-terra and gpt-5.6-luna — get the same twelve app specs, fixed before any results were scored, under seven conditions: vint with the guide in context, vint with only its type declarations, and React 18, Solid, VanJS, Preact with htm and Vue 3. Acceptance tests assert observable DOM behaviour only — text, input values, element identity across reorders, race outcomes, state surviving navigation — and every step settles across two macrotasks so renderers that commit asynchronously are not penalised. On a failure the model gets one fix attempt with the full failure report: pass@1 is first-try correctness, pass@2 second-try diagnosis. At five samples a cell reading 3/5 has a Wilson 95 % interval of 19–88 %, which is worth remembering for most of what follows.
The framework, the guide, the numbered contract, the harness, every task spec and acceptance test, and the raw per-cell result files the tables above are generated from — including the failure reports, which are usually more informative than the pass rates. Change a condition, add an engine, or point the harness at your own spec.
vint on GitHub →Ten small tasks
Ten small-to-trap tasks, five samples per cell. First-try pass rate → pass rate after one fix attempt.
| Condition | Opus 5 | Sonnet 5 | gpt-5.6-terra | gpt-5.6-luna |
|---|---|---|---|---|
| vintwith the guide in context | 49/50 → 50/50 | 48/50 → 50/50 | 49/50 → 50/50 | 48/50 → 49/50 |
| vinttype declarations only | 50/50 | 50/50 | 50/50 | 48/50 → 49/50 |
| React 1818.3 | 50/50 | 50/50 | 50/50 | 50/50 |
| Solid1.9.15 | 50/50 | 50/50 | 48/50 → 49/50 | 49/50 → 50/50 |
| VanJStag functions, no guide | 38/49 → 49/49 | 42/50 → 46/50 | 34/50 → 43/50 | 28/50 → 41/50 |
| Preact with htmReact’s model, no JSX | 50/50 | 50/50 | 50/50 | 49/50 → 50/50 |
| Vue 3runtime + compiler build | 45/45 | 50/50 | 49/50 → 50/50 | 48/50 → 50/50 |
Six of seven conditions sit at 48–50 out of 50 on every engine, vint in both arms; these tasks are too easy to tell the conditions apart, and the type declarations alone carry three of four engines to 50/50. VanJS is the one condition that degrades as the engine gets cheaper — 38 of 49 first-try on Opus 5, 28 of 50 on luna, with the weakest second-attempt recovery of any condition. Its authoring shape is the closest to vint’s, and vint kept that shape while dropping the semantics that fail here.
Removing the guide
The cross-framework table has too many differences in it to isolate anything. The ablation holds everything fixed except the guide, and the difference appears at size: the large task is a three-view project tracker of around 300 lines, with async seed data, nested keyed lists, derived counts crossing views and state surviving navigation.
One ~300-line, three-view app. Five samples per cell.
| Condition | Opus 5 | Sonnet 5 | gpt-5.6-terra | gpt-5.6-luna |
|---|---|---|---|---|
| vintwith the guide in context | 4/5 → 5/5 | 5/5 | 3/5 → 5/5 | 5/5 |
| vinttype declarations only | 3/5 → 5/5 | 3/5 → 4/5 | 3/5 → 4/5 | 1/5 → 4/5 |
| React 18 | 5/5 | 5/5 | 1/5 → 4/5 | 5/5 |
| Solid | 5/5 | 5/5 | 3/5 → 4/5 | 2/5 → 4/5 |
| VanJS | 5/5 | 2/5 → 5/5 | 4/5 | 3/5 → 5/5 |
| Preact with htm | 5/5 | 5/5 | 1/5 → 4/5 | 5/5 |
| Vue 3 | 5/5 | 5/5 | 5/5 | 5/5 |
With the guide, every engine ends at 5/5 and the cheapest reaches it first try. Without it, the same runtime starts at 1–3 of 5 and recovers to 4/5 on three engines of four. Almost all the failures were the same bug — an untracked one-shot read of async state, leaving a heading reading “undefined” permanently — which is what the guide’s first rule is about; with the guide in context it did not occur.
The kanban task
The kanban board is the largest spec after the tracker: async seed data, moves with a timed badge, undo, a filter that hides without removing, edit-in-place with focus retention, a keyboard shortcut. At five samples, vint with the guide, React, VanJS and Preact are all 5/5 on the three more expensive engines; Solid and Vue miss on the first try for seams specific to each — a ref used before the element is in the document, a template ref inside a list read as an element — and mostly recover on the second. The cheapest engine’s cells were re-run at twenty samples, which narrows the intervals enough to read closely, for about a dollar.
gpt-5.6-luna on the kanban task. Twenty samples per condition, first attempt, no fix attempt.
| Condition | First try | Wilson 95 % | Band |
|---|---|---|---|
| Preact with htm | 18/20 | 70–97 % | Usable first try |
| React 18 | 17/20 | 64–95 % | Usable first try |
| VanJS | 17/20 | 64–95 % | Usable first try |
| vintwith the guide in context | 16/20 | 58–92 % | Usable first try |
| vinttype declarations only | 8/20 | 22–61 % | Needs the second attempt |
| Solid | 5/20 | 11–47 % | Mostly wrong first try |
| Vue 3 | 4/20 | 8–42 % | Mostly wrong first try |
The top four rows are indistinguishable at this sample size. The gap is between the two vint rows, which differ only in whether the guide was in context.
What this does not show
- The small tasks ceiling out. Strong models pass nearly everything at that size in every condition, which is why the trap error codes almost never fire.
- Five samples is not much. Everything outside the twenty-sample column is suggestive, and the baselines move between rounds more than vint does with nothing changing on their side.
- These are the tested tasks and engine versions. Nothing here says how any of these frameworks behaves on a real application, or how pleasant any of them is to write by hand.
What it cost
The whole evaluation is about $74 of API spend: roughly Opus 5 $6–29, Sonnet 5 $3–13, terra $5.20, luna $0.20–1. luna is OpenAI’s smallest current-generation model, listed in September 2026 at $0.20 and $1.20 per million input and output tokens against terra’s $2 and $12; its entire two-wave, three-condition run of 151 cells came to about twenty cents, or a fifth of a cent per generated app.
Given the guide in context and a specification to work from, luna scored within one sample of Opus 5 in the same condition on the small tasks and on the large app, and 16/20 on the kanban task against 17/20 for the best condition on that engine. That is the useful result: the model that writes the plan and the model that implements it do not have to be the same one, and the per-token difference between them is a factor of ten or more. We are applying the same constraint to a programming language whose reader is a model rather than a person.
Methodology note. Figures are from the evaluation harness published in the vint repository and are reproducible from the committed result files. Four engines, seven conditions, twelve scored tasks, five samples per cell except where noted; grading is by behavioural acceptance test with one fix attempt per failure; Wilson 95 % intervals accompany every count in the harness output. Model list prices are as published in September 2026. These results concern the tested tasks, conditions and engine versions.