Three-way translation merge: keeping translated pages in sync with an LLM
We recently rebuilt our company website. It exists in English, Swedish and Norwegian and is a static site built with Astro, with pages stored as plain MDX files in a Git repository. As with any multilingual site, you need some method to keep the translated pages in sync between languages. Due to my underlying hubris, defiant curiosity and lack of respect for existing solutions, I wanted to try an idea I’ve had for doing this.
The usual way is to split each page into strings or segments and translate them one by one. Then every language has the same structure, and you correct a translation in the translation tool. I wanted each language to be a whole page that you can edit like any other page. When we edit the English page, we want the Swedish and Norwegian pages to be updated. But that’s very tedious to do by hand. And we can’t re-translate the whole page, because then we might overwrite manual corrections.
What if you gave an LLM the changes made in the source page, and let it edit the existing translated page, the way a coding agent edits a file?
Then it would have the context of the existing text and could perform a semantically equivalent change that wouldn’t overwrite corrected terms. The LLM would get three things:
- the changes made in the source page, as a diff
- the current target page
- an edit tool
Like this:
If you use Git, you may recognize this process. When Git merges two branches, it doesn’t just compare the two versions of a file. It also looks at the last version the two branches have in common, which is called the merge base. That is a three-way merge.
Git needs the base because the two versions alone don’t say who changed what. If a line is in one version and missing in the other, one side may have added it, or the other side may have removed it. With the base, Git can tell which, and it keeps the changes from both sides.
What the LLM does here is similar to a three-way merge. But where an ordinary merge treats both sides alike, here the change only goes one way. Git has an operation for that too, called a cherry-pick. A cherry-pick is a type of three-way merge that takes the change that one commit made and applies it somewhere else:
result = apply(diff(base, theirs), ours)
My idea is the same principle, with an LLM doing the applying:
result = LLM(diff(base, theirs), ours)
Here the base is the old English page, theirs is the new English page, and ours is the current Swedish page. The LLM edits the Swedish page in place instead of producing a new one.
I call this a three-way translation merge: a three-way merge where the two sides are in different languages, and an LLM does the merging.
You can think of each language as a branch of the English page that never merges back. It says the same things, in another language. A sync to one language cherry-picks the new English changes onto that language’s branch, and the base records how far the branch has come. Git can’t do this itself, because it finds the place for a change by matching lines, and on a translated branch none of them match. The LLM finds the place by meaning.
It may be surprising that a diff of the source page is enough to edit a page in another language. But a diff contains the old lines as well as the new ones, and a few unchanged lines around them for context. The LLM uses those to find the place in the target page. For when it needs more than that, it also gets tools to search and read the current source page.
An example
I implemented this idea in my home-built CMS for our company website. Links are at the end of the post, but this post is about the idea and its implications.
To give you an example, this is what it looks like to edit a page in our website admin:
Some background to the video: The word “club” can have two close translations in Swedish: klubb (club) and förening (association). The first machine translation of our page used klubb, because the English page says “club”. But in this context förening is the more appropriate word in Swedish, so we manually corrected the Swedish page to say that instead.
In the video, an editor later changes the heading in the English page, and then the Swedish page is synced:
| English | Swedish | |
|---|---|---|
| Before | Built-in club register | Inbyggt register över föreningar |
| After | Country-specific club register | Landsspecifikt register över föreningar |
The diff shows that only “Built-in” changed, so the LLM changes only that word, and föreningar stays. Translating the whole heading again would have given Landsspecifikt register över klubbar.
Recording the base
To make the diff for a language, you need its base, which is the source page as it was when that language was last synced. There is one base for each target language. Since a target page can change both when someone corrects or localizes it and when it is synced, you can’t know from file timestamps or git history alone when the last sync was. Syncing is an intentional action, so you need to record the base explicitly.
There are many ways to record the base, like marking the sync commits or storing a reference like a commit or blob hash, timestamp or explicit version. All have different tradeoffs. I’ve opted to store each base as a plain file, and update it after each sync. That makes it trivial to detect when a language is out of date: its base differs from the current source page. The diff for the LLM is equally trivial: git diff --no-index base.mdx english.mdx.
You can record the base however you want, though a plain file has one special advantage that we will see. And it’s nice to keep it together with the content.
Corrections to the source page
You might want to change the English page without propagating that change to the other languages. Correcting a spelling mistake in English is one case. To allow this, we can let the author select if changes should propagate (default) or not. If they shouldn’t, we take the changes and patch each target language’s base. After patching, it will be like the target languages have already “seen” that change and it won’t show up in the diff for the LLM, so it isn’t translated.
For you Git nerds, this is like git merge -s ours, which records the other branch’s changes as merged without taking any of them.
Note that if the target language is behind on some other change, the patched base will be a version of the page that was never committed to git. A commit hash can’t point to that, which is one reason to store the base as a file.
The translated page is its own source of truth
Since the LLM has the entire translated page as context, and only edits the page, your corrections to the translated page are kept. Most translation tools save every translated string in a translation memory, and that is also where you would correct something. A three-way translation merge has no translation memory. You correct the translated page itself. The advantage of a translation memory is that the correction is then reused on other pages too, which this approach doesn’t do yet (see problem 3).
Pages that differ
In translation tools that work with strings or segments, every language has the same structure. But since the LLM decides where to edit based on meaning, rather than line numbers, segment ids or translation keys, language versions no longer need to have the same structure. You might want a large banner advertising Vipps payments on the Norwegian page only, or the feature list in a different order on your Spanish page. Changes to the English page would still propagate to those pages.
Here is an example. The samples below are built from our homepage, with the names and prices made up, and I’ve shortened them to fit the figure. The lines that change, and the LLM’s notes, are from a real run with Claude Sonnet 5.5.
In English, the licence price is in one place: the pricing section says that the licence costs SEK 95 per team. The Swedish page is localized, and it says 95 in three places:
- In its pricing section, like the English page.
- In the payments row, which the Swedish page has moved to the top. It has a sentence there that the English page doesn’t have, saying that the payment solution is included in the licence at 95 kr per team.
- In a sentence about another product, which happens to cost 95 kr per month.
Then the price is raised from 95 to 110 in English.
The LLM changed the first two prices and left the third alone, as expected.
Does it work?
This is all still very early, and many questions remain. I’m using it for my small company’s website as an experiment. It works well for us, but that’s a single site.
Using an LLM for translating puts a lot of trust in the model. It basically turns the translation step into a prompt engineering or harness engineering problem. That requires its own methods for reliability. As a start, I’ve built a test suite. It has 50 cases, and in 17 of them the Swedish page differs from the English page, like in the example above. Each case has a list of checks that the Swedish page must pass afterwards. I ran each case three times with Claude Sonnet 5.5, with different inputs for the LLM:
| The LLM gets | Runs passed |
|---|---|
| The diff and the Swedish page, and tools to search and read the English page | 149 of 150 |
| The diff and the Swedish page | 147 of 150 |
| The diff, the Swedish page and the old English page | 150 of 150 |
| Only the current English and Swedish pages, no diff | 113 of 150 |
Without the diff, the LLM can’t tell a change in English from a difference that the Swedish page has on purpose. It made mistakes both ways: it left out real changes, and it removed things that only the Swedish page has. In the price example, one of the three runs removed the Swedish sentence about the licence, because the English page has no such sentence.
With the diff, all the runs that failed were in one case. The Swedish page mentions Klarna in one more place than the English page does, and then English removes every mention of it. The diff shows where English removed it, but not that Klarna is now gone from the whole page. With the old English page or the search tool, the LLM can check that. The runs with the tools cost about 15 percent less than the runs with the old page, since the LLM used the tools in only 14 of the 150 runs.
One run more or less in the table is chance. When I ran the Klarna case 20 times, the LLM got it right 17 times with the old English page, 18 times with the search tool, and 5 times with only the diff.
So in this test, the diff is what makes it work, and the LLM needs a way to look at the source page when a change is about the whole page.
All the results are published, with the diff, the LLM’s notes and the prompt for every run. Note that I’ve only tested it with one model, on short pages, in English and Swedish.
Open problems
1. A mistake can look intentional
If the model gets it wrong, this design can make it hard to see. If the LLM accidentally removes a sentence during a sync, the base is still updated, and from then it’s exactly as if it was removed on purpose. The same goes for a change that the LLM skips. After the sync, that change won’t show up in a diff again. I haven’t seen either happen on our website that I know of. In my implementation every sync is a git commit, so someone could review what it changed, but nothing in the system makes you do it.
One could imagine some form of mechanical checks. For example, we could make the edit tool also require an id for which hunk(s) in the source diff the edit is for. And we could have a stop hook, similar to the ones in Claude Code, that refuses to let the agent stop until it has handled all hunks. Some hunks shouldn’t result in an edit, so there could be a separate tool to let the LLM say that too.
Another option is to make each sync a pull request that someone has to approve.
2. Judgment calls
One could imagine scenarios where the source and target pages “conflict”. If the price is changed to 110 in the English page, and someone has already changed it to 120 in the Swedish page, the LLM has to decide which price the Swedish page should have. In Git this would be a merge conflict.
In my implementation the LLM has a tool for asking a question in these cases. When the translation is run interactively, a person answers. Otherwise the question is logged, and the LLM leaves that part of the page as it is.
3. A correction is only kept locally
When someone corrects “klubb” to “förening” in one sentence, that correction is kept in that file. But the LLM doesn’t know about the correction when it translates new sentences in other files, so “klubb” can pop up in other places. You can easily provide the LLM with a glossary file, but you need to build the glossary file.
I have an idea for this that I haven’t built yet. Since an edit to a translated page is often someone correcting the translation, those edits could be collected and used to update the glossary automatically.
To collect the edits, we could, in addition to the base, also keep a copy or reference to the target page as it was at the last sync. For Swedish, that is the Swedish page as it was right after the LLM last synced it. Anything that differs in the target page since then was changed by a person. If that file says “klubb” and the current Swedish page says “förening”, someone made that correction. The LLM could use a tool to update the glossary file with corrected terms.
Some edits are localizations and not corrections, though, so there needs to be some categorization of edits.
4. Drift
The languages can potentially drift apart over time. With strings, you can check that every string has a translation. With whole pages that are allowed to differ in structure, you can’t, so small mistakes could add up, and the languages could slowly stop saying the same thing. I don’t know how big a problem this is yet.
If it does turn out to be a problem, the categorization from problem 3 above could help. If the LLM sorts the edits into corrections and localizations, maybe we can track the localizations for each page in a structured way. Then it might be possible to run a process that compares the English page with the “unlocalized” Swedish page, to find drift.
5. Languages that are more distant
The pages in the cases are in English and Swedish, which are closely related languages. I haven’t tested languages that are further apart, for example with grammatical cases or a very different word order.
6. Other models
The published runs all used Claude Sonnet 5.5. I don’t know how much the method depends on the model, or how it does with a smaller and cheaper one.
7. Quality of translations
The test suite only checks facts, so it can’t tell good Swedish from stiff Swedish. I also don’t know how a page reads after many small edits, compared with a page that was translated in one go.
Generalizations
Changes in any direction
Taken one step further, if changes are the unit of translation and the pages are their own source of truth, there’s nothing special about propagating changes from English to Swedish. You can just as well edit the Swedish page and let that change propagate to English.
That does need more bookkeeping and has more open problems, and I’ve written about it in a second post: Generalizing the three-way translation merge to any direction.
Beyond translations?
Note that there’s nothing inherent in this idea that limits it to translation between languages. Whenever one document is kept in sync with another that it was derived from, like documentation and the code it describes, you could give an LLM the change and the derived document, instead of producing the derived document again.
The parts are the same. You record the source as it was when the derived document was last updated, you make a diff from that, and the LLM edits the derived document. Edits that people have made to the derived document are kept for the same reason corrections are kept here. Other examples could be a README and the code it documents, a summary of a long report, or a simpler version of a text for another audience. I haven’t tried any of these.
Similar setups
Some translated documentation projects already treat each language as a branch of the English one. Some merge the English changes into each translation with Git, and others record which English version each translation is based on. In all of them, a person translates the changes.
There are also tools that use an LLM to translate only what changed in the source. Most track changes per string or per section and translate those again. While writing this post I did discover that Nextflow’s training material gives the LLM the git diff and the whole existing translation, which is the core idea I describe in this post.
Summary
This post was an exploration of an idea for keeping whole translated pages in sync using LLMs.
- A translated page can be kept in sync with its source page by giving an LLM the changes made in the source page and letting it edit the existing translation. I call it a three-way translation merge. It works like a three-way merge, with the LLM as the merge driver.
- For this you need to record the base, which is the source page as it was at the last sync.
- The translated page is its own source of truth, so corrections made in it are kept.
- The translated pages can differ in structure from the source page.
- The same design can be generalized to let changes go in any direction. That is the subject of a separate post: Generalizing the three-way translation merge to any direction.
In a first test with 50 cases run three times each, 149 of 150 runs passed, although the cases are short and made-up. Without the diff, 113 of 150 passed. The open problems I know of are listed above. The biggest one, I think, is that a mistake by the LLM can look intentional.
I wanted to write down the idea and some of the implications of it. Describing it in text has forced me to think through it more carefully. Hopefully it could be useful for anyone already doing, or considering doing, something similar. Or even if someone is inspired by it.
My implementation is on GitHub as Obelum, if you want to read the code or try it. The CMS is Plinto.
Did you find this interesting? Have you seen someone do this, or something similar? Leave a comment below.