Skip to content

translate rich-text documents between human languages, online or offline

License

Notifications You must be signed in to change notification settings

milahu/translate-richtext

Folders and files

NameName
Last commit message
Last commit date

Latest commit

52d2035 · Mar 16, 2024

History

3 Commits
Mar 4, 2024
Mar 4, 2024
Mar 4, 2024
Mar 4, 2024
Mar 8, 2024
Mar 4, 2024
Mar 16, 2024
Mar 16, 2024

Repository files navigation

translate richtext

translate rich-text documents between human languages, online or offline

solve two conflicting goals:

  1. preserve the original structure of the document, including whitespace, newlines, indents
  2. preserve sentences across structure boundaries like <p><b>hello</b> world.</p>

use cases

  • translate text documents: html, epub, odt, docx, pdf, rtf, latex
  • translate video subtitles: srt, vtt

challenges

preserve sentences

sentences may be broken by

  • newlines in the source text
  • markup tags

this is a problem, because when we feed sentence-parts to translators, then the translators will return worse quality than in the case, where we feed full sentences to translators

align similar texts

our solution is using two translations:

  1. a "splitted" translation
  2. a "joined" translation

the "splitted" translation serves as a "sourcemap", it has the correct positions of sentence-parts, but the translation has worse quality, because sentences are broken into sentence-parts.

the "joined" translation provides the translated sentences, with better quality than the "splitted" translation, but the locations of sentence-parts are lost.

currently, we align the two translations with a "character diff":

git diff --word-diff=color --word-diff-regex=. --no-index \
  $(readlink -f translation.joined.txt) \
  $(readlink -f translation.splitted.txt) |
sed -E $'s/\e\[32m.*?\e\[m//g; s/\e\\[[0-9;:]*[a-zA-Z]//g' |
tail -n +6 >translation.aligned.txt

related

similar projects