← all published runs

Obelum translator evals: peer system

Every language is a peer: each keeps a copy of every language, itself included, and a change in any of them is translated into the others.

claude-sonnet-5-5 · effort medium · commit 907534b (with uncommitted changes) · 2026-10-14 · 53 cases × 3 runs · permanent link

Each case is a page in two or three languages, all in sync. Then someone changes one language, and the translator brings another language up to date with an LLM. The result is checked with plain string tests (a text must or must not appear, nothing outside given lines may change) written to pass for any reasonable wording. Every case has a hand-written reference answer, which a test holds to every check.

This is a snapshot of 3 runs per case, not a statistic. The checks test facts, not how good the language is. Names of products, partners, places and people in the cases are made up, as are their prices. Card, Klarna, Swish, PayPal and Apple Pay are real payment methods. The cases and the runner are in evals/.

About the baseline. The nodiff column is the same translator, model, effort, tools and glossary, given only the current page of each language: no diff and no earlier version. Its prompt is the diff prompt with what depends on the diff rewritten, and asks it to bring the target up to date with the current pages. The diff prompt was chosen among several versions by running these cases. The baseline prompt was written once and not changed, so the comparison favours the diff version. A case with no target page yet is the same in both.

By group

groupcasesnodiff
basic511/15
structure821/24
locality39/9
format712/21
multi-source49/12
new13/3
big411/12
fix47/12
divergent1725/51

basic

One thing changed in a target shaped like the source.

casewhat it testsnodiff$ / runs / run
number A price changes in one sentence. ✓✓✓ $0.010319.2
add-section A new section is added between two others. ✓✓✓ $0.00765.0
remove-section A whole section is removed. ✗✗✗ $0.00463.4
rewrite A paragraph is rewritten with new facts; none of the old wording survives. ✗✓✓ $0.01145.9
glossary A new sentence uses terms the host's glossary decides. ✓✓✓ $0.011719.9

structure

The target is shaped differently from the source: a section of its own, a split paragraph, merged list items, reordered sections, a table for a list.

casewhat it testsnodiff$ / runs / run
target-extra-section The target has a section no other language has, right after the paragraph that changes. ✓✓✓ $0.00785.2
localized-in-changed-line The changed line holds a localization (PayPal became Swish); the change must keep it. ✗✗✗ $0.00826.0
extra-sentence-in-changed-paragraph The target's version of the changed paragraph has a sentence of its own. ✓✓✓ $0.00926.9
split-paragraph The target split one paragraph into two; the change touches both halves. ✓✓✓ $0.00975.5
merged-list-items The target merged two list items into one; the source adds an item. ✓✓✓ $0.00794.4
reordered-sections The target has two sections in the other order; the change is in one of them. ✓✓✓ $0.00745.0
list-became-table The target shows as a table what the source keeps as a list; the source changes a row and adds one. ✓✓✓ $0.00744.8
target-lacks-section The target left out a section on purpose; the source changes something inside it. Either stay without it, or add it whole; not half. ✓✓✓ $0.00945.0

locality

Finding the place: one change in a long page, blocks that read the same, scattered changes.

casewhat it testsnodiff$ / runs / run
one-change-long-page One number changes deep in a 24-entry page; another answer has the same number and must not change. ✓✓✓ $0.00784.7
identical-blocks Three courses share identical detail lines; only the middle one's price changes. ✓✓✓ $0.00764.9
scattered-changes Four small changes spread across a 24-entry page. ✓✓✓ $0.01336.2

format

Front matter, MDX props, code blocks, tables, a moved section, a rewrap, CRLF line endings.

casewhat it testsnodiff$ / runs / run
frontmatter Front matter: the title is text, the date is data, and the target's slug is its own. ✗✗✗ $0.00443.4
mdx-props MDX: a text prop and an image path change; the target's localized href and the ids stay. ✗✗✗ $0.01288.1
code-block A command in a code block changes; the target's translated comment in the block stays. ✓✓✓ $0.00614.8
table-row A row is added to a table. ✓✓✓ $0.00774.9
moved-section The source moves a section without changing a word; the target should move it too, not copy it. ✗✗✗ $0.00423.1
reflow The source only rewraps a paragraph. Nothing to translate; the target's words must not change. ✓✓✓ $0.00623.6
crlf The target uses CRLF line endings, the source LF; edits must keep CRLF. ✓✓✓ $0.00965.9

multi-source

More than one language has changed (two siblings, siblings that disagree, a sibling the target never saw), or the target has edits of its own.

casewhat it testsnodiff$ / runs / run
two-sources en changed the fee and de changed the opening hours; the target needs both. ✗✗✗ $0.01337.6
target-own-fix The target fixed a phrase itself since its last sync; a sibling then changed the same paragraph. ✓✓✓ $0.00916.4
new-sibling A de translation appeared that the target has never seen, shown whole, beside a small en change. Only the en change needs translating. ✓✓✓ $0.00965.1
siblings-disagree en and de changed the same fee to different amounts, and nothing says which is right. The translator cannot settle it: it keeps the old fee and raises the conflict, for a person to answer. ✓✓✓ $0.01186.4

new

A target that does not exist yet.

casewhat it testsnodiff$ / runs / run
from-scratch The target does not exist yet; it is written whole from the source. ✓✓✓ $0.00483.2

big

Pages at real size, a few hundred lines, most of them repeated somewhere.

casewhat it testsnodiff$ / runs / run
catalogue-one-price One price changes in a 30-course catalogue where the same detail lines, and the same price, repeat in ten blocks. ✓✓✓ $0.02224.8
catalogue-diverged The Swedish catalogue is ordered by level, not boat, and has an intro and a course of its own; en changes one course's day and drops another course. ✓✗✓ $0.02147.4
schedule-one-cell One cell changes in a 56-row table whose rows differ only by day, time and activity. ✓✓✓ $0.01116.4
handbook-scattered A 400-line handbook (membership, catalogue, FAQ) where the Swedish has its own section and orders the catalogue its own way; four changes far apart. ✓✓✓ $0.03439.0

fix

The target fixed something itself, then another language changed the same line or sentence.

casewhat it testsnodiff$ / runs / run
same-line-reworded sv added local payment methods to a line as a fix; en then reworded that line (dropping the doubled "card") and moved it up. The reword and the local methods both belong in the result. ✗✗✗ $0.00452.9
localized-then-line-extended sv replaced PayPal with Swish as a fix; en then added invoice to the same line. Swish stays, invoice arrives, PayPal does not come back. ✓✗✗ $0.00666.0
wording-then-sentence-extended sv rephrased a sentence as a fix ("Medlemsavgiften är"); en then changed the amount in that sentence and added to it. The new facts arrive; the chosen phrasing should survive. ✓✓✓ $0.00734.2
far-from-change Control: sv fixed one paragraph, en changed another. Both modes should pass. ✓✓✓ $0.00915.2

divergent

A real product homepage, with every product, partner, place and person name made up. Its Swedish version differs from the English on purpose: a streaming partner only Sweden promotes, payments first, a case study further down.

casewhat it testsnodiff$ / runs / run
en-edits-shared-row en extends the support row. Swedish keeps that row in another place, has a Globex Live row of its own and payments first; the change lands in the support row and nothing Swedish moves. ✓✓✗ $0.02138.6
en-adds-row en adds a referee row after the app row. Swedish gets it, after its own app row, with its ids, and keeps its own order and Globex Live row. ✓✓✓ $0.02067.3
en-adds-row-localized-link Same added row, checked for one convention of this site: every Swedish link starts /sv/. The row's link should too; the prompt's "do not change URLs" pulls the other way. ✓✓✓ $0.01749.2
en-removes-row en drops the Fjordby case study, which Swedish keeps further down. It goes from Swedish too, and nothing else moves. ✗✗✗ $0.025111.4
en-extends-list-sentence en adds QR tickets to the features sentence. Swedish leads that sentence with Globex Live; the new feature arrives and the Swedish lead stays. ✓✓✓ $0.01817.4
en-edits-moved-row en adds Apple Pay to the payments row, which Swedish has moved to the top. The change lands there and the row stays first. ✓✓✓ $0.01866.5
one-change-two-places-price en raises the licence price in its Pricing column, the only place English states it. Swedish states it there and in its payments row at the top; both change. Acme Clubs's own 95 is another price and stays. ✓✗✓ $0.02088.2
one-change-two-places-rename en renames its AI scheduler (SmartSchedule → AutoPlan) in the AI row, the only place English names it. Swedish names it there and in its features sentence; both change. "Smartare schemaläggning" is not the name and stays. ✗✓✗ $0.02379.6
sv-edits-shared-row sv extends the AI row. English gets the change, in its own order, and nothing of Sweden's: no Globex Live, no payments-first. ✗✗✗ $0.02437.3
sv-edits-own-row sv edits its Globex Live row, which English never had. Nothing in English changes. ✗✗✗ $0.02196.7
en-promotes-what-sv-already-promotes en moves its payments row up the page. Swedish already has payments first, so it already does what the change is for: nothing to do. ✗✓✗ $0.01907.3
en-adds-what-sv-already-says en adds that support answers in Swedish, Norwegian and English. Swedish already says so, in the same row: nothing to do. ✗✓✓ $0.01545.4
en-edits-en-only-row en changes its webinar row, which only English has (webinars in English, for organisers abroad). Swedish has nothing it applies to. ✗✗✗ $0.02218.8
en-fixes-english-typo en fixes a typo of its own ("you account" → "your account"). The Swedish never had it: nothing to do. ✓✗✓ $0.01515.7
en-drops-klarna-everywhere en drops Klarna everywhere it mentions it (the payments row and the intro), so Klarna is no longer offered. Swedish loses every Klarna a reader sees, the one in its own features sentence too, while the kort-och-klarna URL and the image path, which are not text, stay. ✗✗✗ $0.045111.8
en-drops-klarna-from-intro-only The twin of en-drops-klarna-everywhere: en shortens the intro so that it no longer lists payment methods, and still offers Klarna in the payments row. Swedish shortens its intro too, and keeps every other Klarna, the one in its own features sentence included. The diff looks much the same as when Klarna is dropped for good; only the rest of the English page tells them apart. ✗✗✗ $0.01325.5
never-synced-keeps-own Swedish has never synced (first adoption): its own order, a Globex Live row only it has, and no East Africa row. It gains the East Africa row and keeps everything of its own. ✓✓✓ $0.02869.8