samples/machine_translation.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Translation Tutorial\n",
    "\n",
    "Machine provides a general framework for machine translation engines. It currently provides implementations for rule-based MT, statistical MT (SMT), and neural MT (NMT). All MT engines implement the same interfaces, which provides a high level of extensibility for calling applications.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Nito.AsyncEx, 5.1.2</span></li><li><span>SIL.Scripture, 12.0.1</span></li><li><span>Thot, 3.4.1</span></li></ul></div></div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#r \"nuget:SIL.Scripture,12.0.1\"\n",
    "#r \"nuget:Thot\"\n",
    "#r \"nuget:Nito.AsyncEx\"\n",
    "#r \"../src/SIL.Machine/bin/Debug/netstandard2.0/SIL.Machine.dll\"\n",
    "#r \"../src/SIL.Machine.Morphology.HermitCrab/bin/Debug/netstandard2.0/SIL.Machine.Morphology.HermitCrab.dll\"\n",
    "#r \"../src/SIL.Machine.Translation.Thot/bin/Debug/netstandard2.0/SIL.Machine.Translation.Thot.dll\"\n",
    "\n",
    "void Write(string text)\n",
    "{\n",
    "    Console.Write(text);\n",
    "}\n",
    "\n",
    "void WriteLine(string text = \"\")\n",
    "{\n",
    "    Console.Write(text + \"\\n\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical Machine Translation\n",
    "\n",
    "Machine provides a phrase-based statistical machine translation engine that is based on the [Thot](https://github.com/sillsdev/thot) library. The SMT engine implemented in Thot is unique, because it supports incremental training and interactive machine translation (IMT). Let's start by training an SMT model. MT models implement the `ITranslationModel` interface. SMT models are trained using a parallel text corpus, so the first step is to create a `ParallelTextCorpus`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "using SIL.Machine.Corpora;\n",
    "using SIL.Machine.Tokenization;\n",
    "\n",
    "var sourceCorpus = new TextFileTextCorpus(\"data/sp.txt\");\n",
    "var targetCorpus = new TextFileTextCorpus(\"data/en.txt\");\n",
    "var parallelCorpus = sourceCorpus.AlignRows(targetCorpus);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trainers are responsible for training MT models. A trainer can be created either using the constructor or using the `CreateTrainer` method on the `ITranslationModel` interface. Creating a trainer by constructor is useful if you are training a new model. The `CreateTrainer` method is useful when you are retraining an existing model. In this example, we are going to construct the trainer directly. Word alignment is at the core of SMT. In this example, we are going to use HMM for word alignment.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model... done.\n",
      "Saving model... done.\n"
     ]
    }
   ],
   "source": [
    "using System.IO;\n",
    "using SIL.Machine.Translation.Thot;\n",
    "using SIL.Machine.Utils;\n",
    "\n",
    "var tokenizer = new LatinWordTokenizer();\n",
    "Directory.CreateDirectory(\"out/sp-en\");\n",
    "File.Copy(\"data/smt.cfg\", \"out/sp-en/smt.cfg\", overwrite: true);\n",
    "{\n",
    "    using var trainer = new ThotSmtModelTrainer(ThotWordAlignmentModelType.Hmm, parallelCorpus, \"out/sp-en/smt.cfg\")\n",
    "    {\n",
    "        SourceTokenizer = tokenizer,\n",
    "        TargetTokenizer = tokenizer,\n",
    "        LowercaseSource = true,\n",
    "        LowercaseTarget = true\n",
    "    };\n",
    "\n",
    "    Write(\"Training model...\");\n",
    "    await trainer.TrainAsync();\n",
    "    WriteLine($\" done.\");\n",
    "    Write(\"Saving model...\");\n",
    "    await trainer.SaveAsync();\n",
    "    WriteLine($\" done.\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to fully translate a sentence, we need to perform pre-processing steps on the source sentence and post-processing steps on the target translation. Here are the steps to fully translate a sentence:\n",
    "\n",
    "1. Tokenize the source sentence.\n",
    "2. Lowercase the source tokens.\n",
    "3. Translate the sentence.\n",
    "4. Truecase the target tokens.\n",
    "5. Detokenize the target tokens into a sentence.\n",
    "\n",
    "Truecasing is the process of properly capitalizing a lowercased sentence. Luckily, Machine provides a statistical truecaser that can learn the capitalization rules for a language. The next step is train the truecaser model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "using SIL.Machine.Translation;\n",
    "\n",
    "{\n",
    "    var truecaser = new UnigramTruecaser(\"out/sp-en/en.truecase.txt\");\n",
    "    using var trainer = truecaser.CreateTrainer(targetCorpus);\n",
    "    await trainer.TrainAsync();\n",
    "    await trainer.SaveAsync();\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a trained SMT model and a trained truecasing model, we are ready to translate sentences. First, We need to load the SMT model. The model can be used to translate sentences using the `TranslateAsync` method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I would like to book a room until tomorrow.\n"
     ]
    }
   ],
   "source": [
    "var truecaser = new UnigramTruecaser(\"out/sp-en/en.truecase.txt\");\n",
    "var detokenizer = new LatinWordDetokenizer();\n",
    "\n",
    "{   \n",
    "    using var model = new ThotSmtModel(ThotWordAlignmentModelType.Hmm, \"out/sp-en/smt.cfg\")\n",
    "    {\n",
    "        SourceTokenizer = tokenizer,\n",
    "        TargetTokenizer = tokenizer,\n",
    "        TargetDetokenizer = detokenizer,\n",
    "        Truecaser = truecaser,\n",
    "        LowercaseSource = true,\n",
    "        LowercaseTarget = true\n",
    "    };\n",
    "\n",
    "    var result = await model.TranslateAsync(\"Desearía reservar una habitación hasta mañana.\");\n",
    "    WriteLine(result.Translation);\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ThotSmtModel` also supports interactive machine translation. Under this paradigm, the engine assists a human translator by providing translations suggestions based on what the user has translated so far. This paradigm can be coupled with incremental training to provide a model that is constantly learning from translator input. Models and engines must implement the `IInteractiveTranslationModel` and `IInteractiveTranslationEngine` interfaces to support IMT. The IMT paradigm is implemented in the `InteractiveTranslator` class. The `ApproveAsync` method on `InteractiveTranslator` performs incremental training using the current prefix. Suggestions are generated from translations using a class that implements the `ITranslationSuggester` interface.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Source: Hablé con recepción.\n",
      "Suggestion: [With reception]\n",
      "Suggestion: I spoke [with reception]\n",
      "Suggestion: I spoke with reception. []\n",
      "\n",
      "Source: Hablé hasta cinco en punto.\n",
      "Suggestion: [I spoke until five o'clock]\n",
      "Suggestion: I spoke until five o'clock. []\n"
     ]
    }
   ],
   "source": [
    "var suggester = new PhraseTranslationSuggester();\n",
    "string GetCurrentSuggestion(InteractiveTranslator translator)\n",
    "{\n",
    "    var suggestion = suggester.GetSuggestions(1, translator).FirstOrDefault();\n",
    "    var suggestionText = suggestion is null ? \"\" : detokenizer.Detokenize(suggestion.TargetWords);\n",
    "    if (translator.Prefix.Length == 0)\n",
    "        suggestionText = suggestionText.Capitalize();\n",
    "    var prefixText = translator.Prefix.Trim();\n",
    "    if (prefixText.Length > 0)\n",
    "        prefixText = prefixText + \" \";\n",
    "    return $\"{prefixText}[{suggestionText}]\";\n",
    "}\n",
    "\n",
    "{\n",
    "    using var model = new ThotSmtModel(ThotWordAlignmentModelType.Hmm, \"out/sp-en/smt.cfg\")\n",
    "    {\n",
    "        SourceTokenizer = tokenizer,\n",
    "        TargetTokenizer = tokenizer,\n",
    "        TargetDetokenizer = detokenizer,\n",
    "        Truecaser = truecaser,\n",
    "        LowercaseSource = true,\n",
    "        LowercaseTarget = true\n",
    "    };\n",
    "    var factory = new InteractiveTranslatorFactory(model)\n",
    "    {\n",
    "        TargetTokenizer = tokenizer,\n",
    "        TargetDetokenizer = detokenizer\n",
    "    };\n",
    "\n",
    "    var sourceSentence = \"Hablé con recepción.\";\n",
    "    WriteLine($\"Source: {sourceSentence}\");\n",
    "    var translator = await factory.CreateAsync(sourceSentence);\n",
    "\n",
    "    var suggestion = GetCurrentSuggestion(translator);\n",
    "    WriteLine($\"Suggestion: {suggestion}\");\n",
    "\n",
    "    translator.AppendToPrefix(\"I spoke \");\n",
    "    suggestion = GetCurrentSuggestion(translator);\n",
    "    WriteLine($\"Suggestion: {suggestion}\");\n",
    "\n",
    "    translator.AppendToPrefix(\"with reception.\");\n",
    "    suggestion = GetCurrentSuggestion(translator);\n",
    "    WriteLine($\"Suggestion: {suggestion}\");\n",
    "    await translator.ApproveAsync(alignedOnly: false);\n",
    "    WriteLine();\n",
    "\n",
    "    sourceSentence = \"Hablé hasta cinco en punto.\";\n",
    "    WriteLine($\"Source: {sourceSentence}\");\n",
    "    translator = await factory.CreateAsync(sourceSentence);\n",
    "\n",
    "    suggestion = GetCurrentSuggestion(translator);\n",
    "    WriteLine($\"Suggestion: {suggestion}\");\n",
    "\n",
    "    translator.AppendToPrefix(\"I spoke until five o'clock.\");\n",
    "    suggestion = GetCurrentSuggestion(translator);\n",
    "    WriteLine($\"Suggestion: {suggestion}\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rule-based Machine Translation\n",
    "\n",
    "Machine provides an implementation of a simple, transfer-based MT engine. Transfer-based MT consists of three steps:\n",
    "\n",
    "1. Analysis: source words are segmented into morphemes.\n",
    "2. Transfer: source morphemes are converted to the equivalent target morphemes.\n",
    "3. Synthesis: the target morphemes are combined into target words.\n",
    "\n",
    "The `TransferEngine` class implements this process. HermitCrab, a rule-based morphological parser, can be used to perform the analysis and synthesis steps. HermitCrab parser implementation is provided in the [SIL.Machine.Morphology.HermitCrab](https://www.nuget.org/packages/SIL.Machine.Morphology.HermitCrab/) package. In this example, the transfer is performed using simple gloss matching.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "God created the world.\n"
     ]
    }
   ],
   "source": [
    "using SIL.Machine.Morphology.HermitCrab;\n",
    "\n",
    "var hcTraceManager = new TraceManager();\n",
    "\n",
    "Language srcLang = XmlLanguageLoader.Load(\"data/sp-hc.xml\");\n",
    "var srcMorpher = new Morpher(hcTraceManager, srcLang);\n",
    "\n",
    "Language trgLang = XmlLanguageLoader.Load(\"data/en-hc.xml\");\n",
    "var trgMorpher = new Morpher(hcTraceManager, trgLang);\n",
    "\n",
    "var transferer = new SimpleTransferer(new GlossMorphemeMapper(trgMorpher));\n",
    "\n",
    "{\n",
    "    using var transferEngine = new TransferEngine(srcMorpher, transferer, trgMorpher)\n",
    "    {\n",
    "        SourceTokenizer = tokenizer,\n",
    "        TargetDetokenizer = detokenizer,\n",
    "        Truecaser = truecaser,\n",
    "        LowercaseSource = true\n",
    "    };\n",
    "\n",
    "    var result = await transferEngine.TranslateAsync(\"Dios creó el mundo.\");\n",
    "    WriteLine(result.Translation.Capitalize());\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hybrid Machine Translation\n",
    "\n",
    "Machine includes a hybrid machine translation approach that allows you to merge the translation results from a rule-based engine and data-driven engine. The translation from the data-drive engine is the base translation. If there are any words/phrases in the base translation that have a low score, then they will be replaced by the translations from the rule-based engine. This hybrid approach is implemented in the `HybridTranslationEngine` class.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SMT: Please make out two cuentas.\n",
      "Transfer: Por favor, haga dos bills.\n",
      "Hybrid: Please make out two bills.\n"
     ]
    }
   ],
   "source": [
    "{\n",
    "    using var smtModel = new ThotSmtModel(ThotWordAlignmentModelType.Hmm, \"out/sp-en/smt.cfg\")\n",
    "    {\n",
    "        SourceTokenizer = tokenizer,\n",
    "        TargetTokenizer = tokenizer,\n",
    "        TargetDetokenizer = detokenizer,\n",
    "        Truecaser = truecaser,\n",
    "        LowercaseSource = true,\n",
    "        LowercaseTarget = true\n",
    "    };\n",
    "\n",
    "    using var transferEngine = new TransferEngine(srcMorpher, transferer, trgMorpher)\n",
    "    {\n",
    "        SourceTokenizer = tokenizer,\n",
    "        TargetDetokenizer = detokenizer,\n",
    "        Truecaser = truecaser,\n",
    "        LowercaseSource = true\n",
    "    };\n",
    "\n",
    "    using var hybridEngine = new HybridTranslationEngine(smtModel, transferEngine)\n",
    "    {\n",
    "        TargetDetokenizer = detokenizer\n",
    "    };\n",
    "\n",
    "    var sourceSentence = \"Por favor, haga dos cuentas.\";\n",
    "    var result = await smtModel.TranslateAsync(sourceSentence);\n",
    "    WriteLine($\"SMT: {result.Translation.Capitalize()}\");\n",
    "\n",
    "    result = await transferEngine.TranslateAsync(sourceSentence);\n",
    "    WriteLine($\"Transfer: {result.Translation.Capitalize()}\");\n",
    "\n",
    "    result = await hybridEngine.TranslateAsync(sourceSentence);\n",
    "    WriteLine($\"Hybrid: {result.Translation.Capitalize()}\");\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "C#"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}