← all published runs

Obelum translator evals: single source

One language is the source; each other language keeps a copy of it only. A case runs here when one language other than the target changes, and that language is the source; edits to the target itself are plain edits. Cases where two languages change have no single-source form and are left out.

claude-sonnet-5-5 · effort medium · commit 6629f4a · 2026-10-10 · 49 cases × 3 runs · permanent link

Each case is a page in two or three languages, all in sync. Then someone changes one language, and the translator brings another language up to date with an LLM. The result is checked with plain string tests (a text must or must not appear, nothing outside given lines may change) written to pass for any reasonable wording. Every case has a hand-written reference answer, which a test holds to every check.

This is a snapshot of 3 runs per case, not a statistic. The checks test facts, not how good the language is. Names of products, partners, places and people in the cases are made up, as are their prices. Card, Klarna, Swish, PayPal and Apple Pay are real payment methods. The cases and the runner are in evals/.

About the baseline. The nodiff column is the same translator, model, effort, tools and glossary, given only the current page of each language: no diff and no earlier version. Its prompt is the diff prompt with what depends on the diff rewritten, and asks it to bring the target up to date with the current pages. The diff prompt was chosen among several versions by running these cases. The baseline prompt was written once and not changed, so the comparison favours the diff version. A case with no target page yet is the same in both.

By group

groupcaseshunksnodiff
basic515/1512/15
structure824/2424/24
locality39/99/9
format721/2112/21
multi-source13/33/3
new13/33/3
big412/1210/12
fix412/128/12
divergent1648/4823/48

basic

One thing changed in a target shaped like the source.

casewhat it testshunksnodiff
number A price changes in one sentence. ✓✓✓✓✓✓
add-section A new section is added between two others. ✓✓✓✓✓✓
remove-section A whole section is removed. ✓✓✓✗✗✗
rewrite A paragraph is rewritten with new facts; none of the old wording survives. ✓✓✓✓✓✓
glossary A new sentence uses terms the host's glossary decides. ✓✓✓✓✓✓

structure

The target is shaped differently from the source: a section of its own, a split paragraph, merged list items, reordered sections, a table for a list.

casewhat it testshunksnodiff
target-extra-section The target has a section no other language has, right after the paragraph that changes. ✓✓✓✓✓✓
localized-in-changed-line The changed line holds a localization (PayPal became Swish); the change must keep it. ✓✓✓✓✓✓
extra-sentence-in-changed-paragraph The target's version of the changed paragraph has a sentence of its own. ✓✓✓✓✓✓
split-paragraph The target split one paragraph into two; the change touches both halves. ✓✓✓✓✓✓
merged-list-items The target merged two list items into one; the source adds an item. ✓✓✓✓✓✓
reordered-sections The target has two sections in the other order; the change is in one of them. ✓✓✓✓✓✓
list-became-table The target shows as a table what the source keeps as a list; the source changes a row and adds one. ✓✓✓✓✓✓
target-lacks-section The target left out a section on purpose; the source changes something inside it. Either stay without it, or add it whole; not half. ✓✓✓✓✓✓

locality

Finding the place: one change in a long page, blocks that read the same, scattered changes.

casewhat it testshunksnodiff
one-change-long-page One number changes deep in a 24-entry page; another answer has the same number and must not change. ✓✓✓✓✓✓
identical-blocks Three courses share identical detail lines; only the middle one's price changes. ✓✓✓✓✓✓
scattered-changes Four small changes spread across a 24-entry page. ✓✓✓✓✓✓

format

Front matter, MDX props, code blocks, tables, a moved section, a rewrap, CRLF line endings.

casewhat it testshunksnodiff
frontmatter Front matter: the title is text, the date is data, and the target's slug is its own. ✓✓✓✗✗✗
mdx-props MDX: a text prop and an image path change; the target's localized href and the ids stay. ✓✓✓✗✗✗
code-block A command in a code block changes; the target's translated comment in the block stays. ✓✓✓✓✓✓
table-row A row is added to a table. ✓✓✓✓✓✓
moved-section The source moves a section without changing a word; the target should move it too, not copy it. ✓✓✓✗✗✗
reflow The source only rewraps a paragraph. Nothing to translate; the target's words must not change. ✓✓✓✓✓✓
crlf The target uses CRLF line endings, the source LF; edits must keep CRLF. ✓✓✓✓✓✓

multi-source

More than one language has changed (two siblings, siblings that disagree, a sibling the target never saw), or the target has edits of its own.

casewhat it testshunksnodiff
target-own-fix The target fixed a phrase itself since its last sync; a sibling then changed the same paragraph. ✓✓✓✓✓✓

new

A target that does not exist yet.

casewhat it testshunksnodiff
from-scratch The target does not exist yet; it is written whole from the source. ✓✓✓✓✓✓

big

Pages at real size, a few hundred lines, most of them repeated somewhere.

casewhat it testshunksnodiff
catalogue-one-price One price changes in a 30-course catalogue where the same detail lines, and the same price, repeat in ten blocks. ✓✓✓✓✓✓
catalogue-diverged The Swedish catalogue is ordered by level, not boat, and has an intro and a course of its own; en changes one course's day and drops another course. ✓✓✓✓✗✗
schedule-one-cell One cell changes in a 56-row table whose rows differ only by day, time and activity. ✓✓✓✓✓✓
handbook-scattered A 400-line handbook (membership, catalogue, FAQ) where the Swedish has its own section and orders the catalogue its own way; four changes far apart. ✓✓✓✓✓✓

fix

The target fixed something itself, then another language changed the same line or sentence.

casewhat it testshunksnodiff
same-line-reworded sv added local payment methods to a line as a fix; en then reworded that line (dropping the doubled "card") and moved it up. The reword and the local methods both belong in the result. ✓✓✓✗✗✗
localized-then-line-extended sv replaced PayPal with Swish as a fix; en then added invoice to the same line. Swish stays, invoice arrives, PayPal does not come back. ✓✓✓✓✓✗
wording-then-sentence-extended sv rephrased a sentence as a fix ("Medlemsavgiften är"); en then changed the amount in that sentence and added to it. The new facts arrive; the chosen phrasing should survive. ✓✓✓✓✓✓
far-from-change Control: sv fixed one paragraph, en changed another. Both modes should pass. ✓✓✓✓✓✓

divergent

A real product homepage, with every product, partner, place and person name made up. Its Swedish version differs from the English on purpose: a streaming partner only Sweden promotes, payments first, a case study further down.

casewhat it testshunksnodiff
en-edits-shared-row en extends the support row. Swedish keeps that row in another place, has a Globex Live row of its own and payments first; the change lands in the support row and nothing Swedish moves. ✓✓✓✗✓✓
en-adds-row en adds a referee row after the app row. Swedish gets it, after its own app row, with its ids, and keeps its own order and Globex Live row. ✓✓✓✓✓✓
en-adds-row-localized-link Same added row, checked for one convention of this site: every Swedish link starts /sv/. The row's link should too; the prompt's "do not change URLs" pulls the other way. ✓✓✓✓✓✓
en-removes-row en drops the Fjordby case study, which Swedish keeps further down. It goes from Swedish too, and nothing else moves. ✓✓✓✗✗✗
en-extends-list-sentence en adds QR tickets to the features sentence. Swedish leads that sentence with Globex Live; the new feature arrives and the Swedish lead stays. ✓✓✓✗✓✗
en-edits-moved-row en adds Apple Pay to the payments row, which Swedish has moved to the top. The change lands there and the row stays first. ✓✓✓✓✓✓
one-change-two-places-price en raises the licence price in its Pricing column, the only place English states it. Swedish states it there and in its payments row at the top; both change. Acme Clubs's own 95 is another price and stays. ✓✓✓✓✗✓
one-change-two-places-rename en renames its AI scheduler (SmartSchedule → AutoPlan) in the AI row, the only place English names it. Swedish names it there and in its features sentence; both change. "Smartare schemaläggning" is not the name and stays. ✓✓✓✗✗✗
sv-edits-shared-row sv extends the AI row. English gets the change, in its own order, and nothing of Sweden's: no Globex Live, no payments-first. ✓✓✓✗✗✗
sv-edits-own-row sv edits its Globex Live row, which English never had. Nothing in English changes. ✓✓✓✗✗✗
en-promotes-what-sv-already-promotes en moves its payments row up the page. Swedish already has payments first, so it already does what the change is for: nothing to do. ✓✓✓✓✗✗
en-adds-what-sv-already-says en adds that support answers in Swedish, Norwegian and English. Swedish already says so, in the same row: nothing to do. ✓✓✓✓✓✓
en-edits-en-only-row en changes its webinar row, which only English has (webinars in English, for organisers abroad). Swedish has nothing it applies to. ✓✓✓✗✗✗
en-fixes-english-typo en fixes a typo of its own ("you account" → "your account"). The Swedish never had it: nothing to do. ✓✓✓✓✓✗
en-drops-klarna-everywhere en drops Klarna everywhere it mentions it (the payments row and the intro), so Klarna is no longer offered. Swedish loses every Klarna a reader sees, the one in its own features sentence too, while the kort-och-klarna URL and the image path, which are not text, stay. ✓✓✓✗✗✗
never-synced-keeps-own Swedish has never synced (first adoption): its own order, a Globex Live row only it has, and no East Africa row. It gains the East Africa row and keeps everything of its own. ✓✓✓✓✓✓