We gave nine AI models a language we had just invented.

Nine model configurations were given an invented language and asked to translate a story written in it. Several did so accurately. Others changed what happened.

Every word they needed was defined in the document they were given. There was nothing to retrieve and no outside knowledge to apply. The errors that came back are the useful part, because we are trying to write a specification a model has to apply rather than read, and these are the ways such a document fails.

The experiment

We used AI to create a small invented language called Nemi. Each word is a pair of Greek Unicode letters, used as arbitrary labels with no connection to their Greek names or values. The document we assembled contains a dictionary, eight grammar rules, and a story written entirely in Nemi, with no English translation of it anywhere in the file.

Part of the experiment	What we supplied
Vocabulary	150 entries; 112 of them used in the story
Representation	Two Greek Unicode characters per word
Grammar	Eight rules: word order, tense, negation, number, possession, questions, references
Story	Ten paragraphs; 559 Nemi word occurrences including the title
Prompt	“can you translate this story.” The document itself asks for natural English.
Withheld	The English composition notes and the reference translation

Part of the dictionary, including the entries that matter later:

Clauses run subject–tense–negation–modal–verb–object, with optional elements omitted. Descriptive words follow their noun. There are no gender distinctions and no verb inflections. Unmarked count nouns are singular, and a separate marker indicates the plural. So:

αα εβ εε νδ αδ

I · only · past · ring · it

“I only rang it.”

The story, Seed and Voice, is a flood narrative. A traveller arrives at a village carrying a bowl with a seed in it. The village lost its bell in an earlier flood, when the caretaker used the bell’s rope during a rescue. The traveller and a child plant the seed; the empty bowl can then be struck with a stone. When the river rises again the child rings the bowl to warn the houses, and the villagers reach a hill. The seed later becomes a tree, which the child names “Voice.”

The story was built so that a model could recover its general shape while still getting details wrong: the difference between a past rescue and a future intention, between the main child and a second, smaller one, between “only” and negation, and the number of the bird at the end.

What came back

One translation was collected from each of nine model configurations. The names are the labels recorded during the sessions; snapshot identifiers were not captured. Speed is a qualitative observation, not a measurement, and the interfaces and output lengths were not normalised. Fidelity is a comparison against the dictionary, the grammar and the source story, not a score.

Model as recorded	Reasoning	Observed speed	Fidelity	What the translation showed
Haiku 4.5	Extended	Very fast	Substantially wrong	Omissions, invented dialogue, incorrect tense and number, reversed meaning
Sonnet 5	Medium	Slow	Faithful	Faithful overall; pluralised the final bird
Opus 5	Medium	Fast	Faithful	Preserved the grammatical and narrative distinctions
Fable 5	Medium	Slow	Faithful	Faithful overall; defended pluralising the final bird
Luna 5.6	Medium	Fast	Substantially wrong	Vocabulary substitutions; omitted a rescue action
Terra 5.6	Medium	Slow	Mostly faithful	Changed explicit content and added plausible details
Sol 5.6	High	Extremely slow	Faithful	Faithful, with extensive explanatory notes
Astra 6	High	Slow	Faithful	Faithful; noted the genuine ambiguity about timing
Grok ExpertCurrent model in the interface	—	Slow	Faithful	More literal English, accurate decoding

Faithful — the distinctions that mattered survived Mostly faithful — changed or added content that reads naturally Substantially wrong — dictionary errors that change what happened

Opus, Sol, Astra and Grok were the strongest group, with Sonnet and Fable close behind. Terra made more consequential changes. Luna and Haiku made substantial errors, with different failure patterns. Time spent did not track with accuracy: Haiku ran with extended reasoning and still landed in the weakest band, and Opus was both fast and accurate.

How the translations went wrong

Three failure patterns are worth separating, because each one produces output that reads as finished work.

A wrong symbol that is still a valid symbol

Several of Luna’s substitutions can be traced to other entries in the dictionary.

Intended	Source	Luna wrote	Entry for the substituted word
Winter	ιε	Child	ει
Tree	ηζ	Window	ζη
Bell	ζκ	Star	θκ
Voice	κη	Name	κθ

The bell became a star repeatedly: the star had a rope, fell into the river, and was replaced by a new star. The source does not permit a choice here, so these are errors rather than readings. What makes them hard to catch is that a single wrong mapping propagates and stays internally consistent. Once the bell is a star, every later reference agrees with the first mistake.

An inverted meaning

Haiku translated αα εβ εε νδ αδ — “I only rang it” — as “I did not ring it.” The two entries involved are εβ, only, and βε, not. In the source the child acknowledges a limited contribution: the child rang the bowl, and other people opened doors and helped their friends. The translation has the child deny ringing it.

The model then listed the sentence as ambiguous and offered negative readings in its explanation. The word was explicitly defined, so the uncertainty was introduced by the response rather than found in the source. That distinction matters when judging this kind of document: a model can sound careful by discussing ambiguity while failing to check the rule that resolves it. Haiku also translated the plural marker εδ as “two,” which is a separate entry, ββ.

An addition the source did not authorise

Terra’s translation was better than Luna’s, but some of its changes are harder to notice because they sound natural.

Source meaning	Terra’s rendering	What changed
“Can it make a new bell?”	“Can it make a new house?”	The object of the question
“A bird can make a house in the tree.”	“Birds can make homes in a very large tree.”	Singular to plural; size added
The child went to the house beside the first and rang the bowl again	The child went from house to house, ringing it again and again	A specific action became a repeated one

A tree where birds make homes is plausible, and so is a child moving from house to house during an evacuation. Plausibility does not make either part of the supplied text. There is room for natural English in a faithful translation — “shade” for “shadow,” “home” for “house” — but changing an object or adding a property changes the content. Grok is the counterpoint: its English was more literal, including phrases like “the friends,” and it preserved the source more closely than some smoother responses. Prose quality and fidelity needed separate assessments.

The final sentence is a smaller case of the same thing. The grammar states that unmarked count nouns are singular, and the story ends δκ. γε ιζ, ιι εζ λγ. — “Now. In spring, a bird will come.” Sonnet, Fable and Terra all wrote “birds.” Fable described singular and plural as equally faithful despite the rule. There was a real ambiguity nearby, which Astra identified: the story was already set in spring, and the sentence does not say whether the bird arrives later that spring or in a subsequent one. The number was specified; the timing was not.

What this has to do with the language we are building

We are building a programming language whose reader is a model rather than a person. Much of what makes a language pleasant for people exists to help a human parse text, and one step in particular can be removed: the scan that turns a stream of characters into the language’s tokens. The source is written in those tokens directly, which is the same property Nemi has — ζκ is one indivisible word, and nothing has to work out where it begins and ends.

Two things are called tokens here. A language token is a unit of the language: a name, an operator, a literal. A model token is a unit of whatever the model’s tokenizer produces. Our 2,000-token budget is the second kind, and removing the first kind of tokenisation says nothing about the second.

The difficulty is that a model is good at a programming language largely because it has read a great deal of that language, and a language invented this year has no corpus. The bet is that this matters less than it appears. Competence at programming is not mostly a memory of particular languages: a model asked to port a function from Go to Rust will usually do it correctly without recalling a translation it has seen, because it holds the underlying ideas and knows how each language spells them. If the specification supplies the spelling, the existing training should carry the rest.

That is what the Nemi exercise tested. Every concept in the story was one the models already had — a child, a river, a past event, a negation, a plural, a claim narrowed by “only.” Only the symbols were new, and the question was whether the mapping held across a whole document.

The job we want done

The goal is a division of labour: a strong model reads the problem and writes the plan, and a small, fast, inexpensive model implements it. The judgement sits in the plan, and the implementation should be mechanical enough that the cheaper model does not need that judgement again. This only works if the cheaper model does not alter the plan while implementing it.

The experiment did not clear that bar. The two configurations we would use as implementers, Haiku and Luna, were the two that failed. Nemi is not our language, and its symbols were chosen to be compact rather than distinguishable, so this settles nothing about the design itself. It does set the standard: a specification a strong model can follow and a cheap model cannot is not useful for this purpose.

We have built one of these before

vint is a TypeScript UI framework we open-sourced whose primary user is an AI rather than a person: real DOM, no virtual DOM, no JSX, no build step, and a single guide file that ships with the runtime and is meant to be in context whenever the framework is used.

Two things from it carry over. Where vint uses a name from Solid it implements Solid’s behaviour faithfully, so a model’s existing expectation is already correct, and the deliberate divergences are stated at the top of the guide rather than left to be discovered; the guide spends its space on the differences. And the size constraint came from the same reasoning — small enough to resend on every turn without the cost becoming an argument against using it, complete enough that nothing else is needed alongside it.

vint ships with an evaluation, and one of its results bears on this directly. On the cheapest engine tested, on a task written by someone other than the framework’s author, the same runtime scored 8 of 20 first try with only its type declarations and 16 of 20 with the guide in context; React scored 17 of 20 on that engine. The whole evaluation arm for that engine cost about twenty cents in API spend, roughly a fifth of a cent per generated app.

The cheap model needed the guide →

vint on GitHub →

What the specification has to establish

The three failures above are the three that matter in generated code, because each produces something that still runs. A wrong symbol that is nonetheless a valid symbol type-checks, executes and agrees with itself wherever it appears, so neither internal consistency nor a compiler will catch it. An inverted condition is a guard that admits what it was written to refuse. An unauthorised addition is the model writing the idiomatic version of the request rather than the request — an extra retry, a cache, a defaulted argument.

So the specification has to establish the mapping and its limit: that a construct is the operation the model already knows by another name, and also where the familiar default stops applying. “This is the construct you already know” must not become “do everything you usually do with it.”

Compactness also has to be measured at the right level. Two-character words are visually compact, which says nothing about their cost to the model; the explanation needs measuring with the target model’s tokenizer, with the vocabulary, rules, examples and instructions all inside the budget and the task payload counted separately. Correctness needs measuring too, since a shorter representation that causes enough errors to require correction is a poor design even when the first prompt is smaller. Examples are part of that trade: a few contrasts that each change one feature may be worth more than a longer abstract description.

Nemi example	Intended English
αα εε νδ αδ.	I rang it.
αα εε βε νδ αδ.	I did not ring it.
αα εβ εε νδ αδ.	I only rang it.
αα εζ νδ αδ.	I will ring it.

Limits, and what we would do next

This was an exploratory exercise: one response per configuration, qualitative timing, and AI-assisted construction and analysis. It was not a repeated or blinded benchmark, and it does not establish why the models differed. The comparisons are open to inspection through the published dictionary and the quoted errors rather than through a score.

The task was also easier than an unfamiliar natural language, since much of it is dictionary substitution followed by grammatical reconstruction, and we supplied the complete vocabulary rather than requiring it to be inferred. Nor does the exercise show that we taught a model a language in the sense a person learns one; it shows a model applying supplied information while drawing on capabilities it already had.

Our source had flaws. Some prepositional phrases could attach in more than one way, commands were used without an explicit imperative rule, the placement of “only” was underspecified, and the final spring reference was unclear. Those limits do not make ζκ ambiguous between bell and star, or turn a defined singular into an unspecified plural. Both things are true: the document can be improved, and several responses contradicted information it already supplied.

A more controlled follow-up would change one thing at a time.

We would also score outputs at the level of meaning — objects, actions, tense, number, negation, references, omissions, unsupported additions — rather than as an overall impression, which is the same discipline we applied to document extraction, where a single accuracy figure concealed whether a value was still attached to what it described.

Read: A Number Is Only Useful If You Know What It Belongs To →

The last of those follow-ups is the most useful diagnostic. Because this language has a fixed dictionary and simple grammar, software can mechanically replace symbols with their glosses before a model produces English. Comparing that route against direct translation would separate lookup errors from interpretation errors, and would tell us whether a production system should delegate exact symbol conversion to a deterministic component.

Where this leaves us

The strongest responses show that a short, self-contained description can be enough for faithful interpretation of a constrained language. The weaker ones show how fluent output can conceal failures in the supplied mappings, and they came from the configurations we want doing the implementing.

The next step is to spend the budget on clear concept mappings, compositional rules, and examples that separate the mistakes we actually observed, then measure what survives. The requirement is straightforward to state: a small specification whose meaning can be relied on, where a defined word keeps its definition from the top of a document to the bottom.

Take the Nemi challenge yourself →