samples/corpora.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Corpora Tutorial\n",
    "\n",
    "In this notebook, we will demonstrate how to use Machine to load datasets as text corpora."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>SIL.Scripture, 12.0.1</span></li></ul></div></div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#r \"nuget:SIL.Scripture,12.0.1\"\n",
    "#r \"../src/SIL.Machine/bin/Debug/netstandard2.0/SIL.Machine.dll\"\n",
    "\n",
    "void WriteLine(string text = \"\")\n",
    "{\n",
    "    Console.Write(text + \"\\n\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Text Files\n",
    "\n",
    "Let's start with a simple example of loading a set of text files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "using SIL.Machine.Corpora;\n",
    "\n",
    "var corpus = new TextFileTextCorpus(\"data/en_tok.txt\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is easy to iterate through the sentences in the corpus. We simply iterate on the corpus object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Would you mind giving us the keys to the room , please ?\n",
      "I have made a reservation for a quiet , double room with a telephone and a tv for Rosario Cabedo .\n",
      "Would you mind moving me to a quieter room ?\n",
      "I have booked a room .\n",
      "I think that there is a problem .\n",
      "Do you have any rooms with a tv , air conditioning and a safe available ?\n",
      "Would you mind showing us a room with a tv ?\n",
      "Does it have a telephone ?\n",
      "I am leaving on the second at eight in the evening .\n",
      "How much does a single room cost per week ?\n"
     ]
    }
   ],
   "source": [
    "foreach (var row in corpus.Take(10))\n",
    "    WriteLine(row.Text);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Scripture\n",
    "\n",
    "Machine contains classes for loading Scripture in various formats, such as USFM and USX."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### USX\n",
    "\n",
    "USX is a common XML format for Scripture. Let's take a look at how to load a set of USX files. First, we create an instance of the `UsxFileTextCorpus` class. We ensure that the correct verse references are used by loading the versification file for this translation. If a versification is not provided, then the English versification is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "using SIL.Scripture;\n",
    "\n",
    "var versification = Versification.Table.Implementation.Load(\"data/WEB-DBL/release/versification.vrs\", \"web\");\n",
    "var corpus = new UsxFileTextCorpus(\"data/WEB-DBL/release/USX_1\", versification);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's iterate through the corpus. You will notice that each text segment in the corpus has an associated reference. In the case of Scripture, these are `VerseRef` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1: That which was from the beginning, that which we have heard, that which we have seen with our eyes, that which we saw, and our hands touched, concerning the Word of life\n",
      "1JN 1:2: (and the life was revealed, and we have seen, and testify, and declare to you the life, the eternal life, which was with the Father, and was revealed to us);\n",
      "1JN 1:3: that which we have seen and heard we declare to you, that you also may have fellowship with us. Yes, and our fellowship is with the Father and with his Son, Jesus Christ.\n",
      "1JN 1:4: And we write these things to you, that our joy may be fulfilled.\n",
      "1JN 1:5: This is the message which we have heard from him and announce to you, that God is light, and in him is no darkness at all.\n",
      "1JN 1:6: If we say that we have fellowship with him and walk in the darkness, we lie and don’t tell the truth.\n",
      "1JN 1:7: But if we walk in the light as he is in the light, we have fellowship with one another, and the blood of Jesus Christ his Son, cleanses us from all sin.\n",
      "1JN 1:8: If we say that we have no sin, we deceive ourselves, and the truth is not in us.\n",
      "1JN 1:9: If we confess our sins, he is faithful and righteous to forgive us the sins and to cleanse us from all unrighteousness.\n",
      "1JN 1:10: If we say that we haven’t sinned, we make him a liar, and his word is not in us.\n"
     ]
    }
   ],
   "source": [
    "foreach (var row in corpus.Take(10))\n",
    "    WriteLine($\"{row.Ref}: {row.Text}\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also iterate through verses in the corpus by book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN\n",
      "======\n",
      "1:1: That which was from the beginning, that which we have heard, that which we have seen with our eyes, that which we saw, and our hands touched, concerning the Word of life\n",
      "1:2: (and the life was revealed, and we have seen, and testify, and declare to you the life, the eternal life, which was with the Father, and was revealed to us);\n",
      "1:3: that which we have seen and heard we declare to you, that you also may have fellowship with us. Yes, and our fellowship is with the Father and with his Son, Jesus Christ.\n",
      "\n",
      "2JN\n",
      "======\n",
      "1:1: The elder, to the chosen lady and her children, whom I love in truth, and not I only, but also all those who know the truth,\n",
      "1:2: for the truth’s sake, which remains in us, and it will be with us forever:\n",
      "1:3: Grace, mercy, and peace will be with us, from God the Father and from the Lord Jesus Christ, the Son of the Father, in truth and love.\n",
      "\n",
      "3JN\n",
      "======\n",
      "1:1: The elder to Gaius the beloved, whom I love in truth.\n",
      "1:2: Beloved, I pray that you may prosper in all things and be healthy, even as your soul prospers.\n",
      "1:3: For I rejoiced greatly when brothers came and testified about your truth, even as you walk in truth.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "foreach (var text in corpus.Texts)\n",
    "{\n",
    "    WriteLine(text.Id);\n",
    "    WriteLine(\"======\");\n",
    "    foreach (var row in text.Take(3))\n",
    "    {\n",
    "        var verseRef = (VerseRef)row.Ref;\n",
    "        WriteLine($\"{verseRef.Chapter}:{verseRef.Verse}: {row.Text}\");\n",
    "    }\n",
    "    WriteLine();\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Digital Bible Library Bundles\n",
    "\n",
    "Now, let's load a Digital Bible Library (DBL) bundle. A DBL bundle is a zip archive that contains all of the data that you need for a publishable Bible translation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DBL bundle created.\n"
     ]
    }
   ],
   "source": [
    "using System.IO;\n",
    "using System.IO.Compression;\n",
    "\n",
    "Directory.CreateDirectory(\"out\");\n",
    "if (File.Exists(\"out/web.zip\"))\n",
    "    File.Delete(\"out/web.zip\");\n",
    "ZipFile.CreateFromDirectory(\"data/WEB-DBL\", \"out/web.zip\");\n",
    "WriteLine(\"DBL bundle created.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we create a `DblBundleTextCorpus` instance. There is no need to specify versification, because the `DblBundleTextCorpus` class takes care of that for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var corpus = new DblBundleTextCorpus(\"out/web.zip\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can iterate through the corpus just as we did before. All text corpus classes in Machine adhere to the same interface, so it is easy to switch between the various classes. Also, you can see that the verse text is nicely tokenized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1: That which was from the beginning, that which we have heard, that which we have seen with our eyes, that which we saw, and our hands touched, concerning the Word of life\n",
      "1JN 1:2: (and the life was revealed, and we have seen, and testify, and declare to you the life, the eternal life, which was with the Father, and was revealed to us);\n",
      "1JN 1:3: that which we have seen and heard we declare to you, that you also may have fellowship with us. Yes, and our fellowship is with the Father and with his Son, Jesus Christ.\n",
      "1JN 1:4: And we write these things to you, that our joy may be fulfilled.\n",
      "1JN 1:5: This is the message which we have heard from him and announce to you, that God is light, and in him is no darkness at all.\n",
      "1JN 1:6: If we say that we have fellowship with him and walk in the darkness, we lie and don’t tell the truth.\n",
      "1JN 1:7: But if we walk in the light as he is in the light, we have fellowship with one another, and the blood of Jesus Christ his Son, cleanses us from all sin.\n",
      "1JN 1:8: If we say that we have no sin, we deceive ourselves, and the truth is not in us.\n",
      "1JN 1:9: If we confess our sins, he is faithful and righteous to forgive us the sins and to cleanse us from all unrighteousness.\n",
      "1JN 1:10: If we say that we haven’t sinned, we make him a liar, and his word is not in us.\n"
     ]
    }
   ],
   "source": [
    "foreach (var row in corpus.Take(10))\n",
    "    WriteLine($\"{row.Ref}: {row.Text}\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Paratext Projects\n",
    "\n",
    "Another useful text corpus class is `ParatextTextCorpus`. This class is used to load a Paratext project. It properly loads the configured encoding and versification for the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var corpus = new ParatextTextCorpus(\"data/WEB-PT\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's iterate through the segments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1: That which was from the beginning, that which we have heard, that which we have seen with our eyes, that which we saw, and our hands touched, concerning the Word of life\n",
      "1JN 1:2: (and the life was revealed, and we have seen, and testify, and declare to you the life, the eternal life, which was with the Father, and was revealed to us);\n",
      "1JN 1:3: that which we have seen and heard we declare to you, that you also may have fellowship with us. Yes, and our fellowship is with the Father and with his Son, Jesus Christ.\n",
      "1JN 1:4: And we write these things to you, that our joy may be fulfilled.\n",
      "1JN 1:5: This is the message which we have heard from him and announce to you, that God is light, and in him is no darkness at all.\n",
      "1JN 1:6: If we say that we have fellowship with him and walk in the darkness, we lie and don’t tell the truth.\n",
      "1JN 1:7: But if we walk in the light as he is in the light, we have fellowship with one another, and the blood of Jesus Christ his Son, cleanses us from all sin.\n",
      "1JN 1:8: If we say that we have no sin, we deceive ourselves, and the truth is not in us.\n",
      "1JN 1:9: If we confess our sins, he is faithful and righteous to forgive us the sins and to cleanse us from all unrighteousness.\n",
      "1JN 1:10: If we say that we haven’t sinned, we make him a liar, and his word is not in us.\n"
     ]
    }
   ],
   "source": [
    "foreach (var row in corpus.Take(10))\n",
    "    WriteLine($\"{row.Ref}: {row.Text}\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parallel Text Corpora\n",
    "\n",
    "So far we have only dealt with monolingual corpora. For many tasks, such as machine translation, parallel corpora are required. Machine provides a corpus class for combining two monolingual corpora into a parallel corpus."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to create a parallel text corpus, we must first create the source and target monolingual text corpora. Then, we create the parallel corpus using the `AlignRows` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var sourceCorpus = new ParatextTextCorpus(\"data/VBL-PT\");\n",
    "var targetCorpus = new ParatextTextCorpus(\"data/WEB-PT\");\n",
    "var parallelCorpus = sourceCorpus.AlignRows(targetCorpus);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now iterate through the parallel segments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1\n",
      "Source: Esta carta trata sobre la Palabra de vida que existía desde el principio, que hemos escuchado, que hemos visto con nuestros propios ojos y le hemos contemplado, y que hemos tocado con nuestras manos.\n",
      "Target: That which was from the beginning, that which we have heard, that which we have seen with our eyes, that which we saw, and our hands touched, concerning the Word of life\n",
      "1JN 1:2\n",
      "Source: Esta Vida nos fue revelada. La vimos y damos testimonio de ella. Estamos hablándoles de Aquél que es la Vida Eterna, que estaba con el Padre, y que nos fue revelado.\n",
      "Target: (and the life was revealed, and we have seen, and testify, and declare to you the life, the eternal life, which was with the Father, and was revealed to us);\n",
      "1JN 1:3\n",
      "Source: Los que hemos visto y oído eso mismo les contamos, para que también puedan participar de esta amistad junto a nosotros. Esta amistad con el Padre y su Hijo Jesucristo.\n",
      "Target: that which we have seen and heard we declare to you, that you also may have fellowship with us. Yes, and our fellowship is with the Father and with his Son, Jesus Christ.\n",
      "1JN 1:4\n",
      "Source: Escribimos para decirles esto, a fin de que nuestra felicidad sea completa.\n",
      "Target: And we write these things to you, that our joy may be fulfilled.\n",
      "1JN 1:5\n",
      "Source: Este es el mensaje que recibimos de él y que nosotros les declaramos a ustedes: Dios es luz, y no hay ningún vestigio de oscuridad en él.\n",
      "Target: This is the message which we have heard from him and announce to you, that God is light, and in him is no darkness at all.\n"
     ]
    }
   ],
   "source": [
    "foreach (var row in parallelCorpus.Take(5))\n",
    "{\n",
    "    WriteLine($\"{row.Ref}\");\n",
    "    WriteLine($\"Source: {row.SourceText}\");\n",
    "    WriteLine($\"Target: {row.TargetText}\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corpus Processing\n",
    "\n",
    "Often a text corpus must be processed in some way as a part of a AI/ML pipeline. Machine has a set of operations that can be used to process a corpus easily. Lowercasing text is a common pre-processing step, so let's show how to apply the \"lowercase\" operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1: that which was from the beginning, that which we have heard, that which we have seen with our eyes, that which we saw, and our hands touched, concerning the word of life\n",
      "1JN 1:2: (and the life was revealed, and we have seen, and testify, and declare to you the life, the eternal life, which was with the father, and was revealed to us);\n",
      "1JN 1:3: that which we have seen and heard we declare to you, that you also may have fellowship with us. yes, and our fellowship is with the father and with his son, jesus christ.\n",
      "1JN 1:4: and we write these things to you, that our joy may be fulfilled.\n",
      "1JN 1:5: this is the message which we have heard from him and announce to you, that god is light, and in him is no darkness at all.\n",
      "1JN 1:6: if we say that we have fellowship with him and walk in the darkness, we lie and don’t tell the truth.\n",
      "1JN 1:7: but if we walk in the light as he is in the light, we have fellowship with one another, and the blood of jesus christ his son, cleanses us from all sin.\n",
      "1JN 1:8: if we say that we have no sin, we deceive ourselves, and the truth is not in us.\n",
      "1JN 1:9: if we confess our sins, he is faithful and righteous to forgive us the sins and to cleanse us from all unrighteousness.\n",
      "1JN 1:10: if we say that we haven’t sinned, we make him a liar, and his word is not in us.\n"
     ]
    }
   ],
   "source": [
    "var corpus = new ParatextTextCorpus(\"data/WEB-PT\");\n",
    "\n",
    "foreach (var row in corpus.Lowercase().Take(10))\n",
    "    WriteLine($\"{row.Ref}: {row.Text}\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multiple operations can be chained together. Here we will tokenize, lowercase, and normalize the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1: that which was from the beginning , that which we have heard , that which we have seen with our eyes , that which we saw , and our hands touched , concerning the word of life\n",
      "1JN 1:2: ( and the life was revealed , and we have seen , and testify , and declare to you the life , the eternal life , which was with the father , and was revealed to us ) ;\n",
      "1JN 1:3: that which we have seen and heard we declare to you , that you also may have fellowship with us . yes , and our fellowship is with the father and with his son , jesus christ .\n",
      "1JN 1:4: and we write these things to you , that our joy may be fulfilled .\n",
      "1JN 1:5: this is the message which we have heard from him and announce to you , that god is light , and in him is no darkness at all .\n",
      "1JN 1:6: if we say that we have fellowship with him and walk in the darkness , we lie and don’t tell the truth .\n",
      "1JN 1:7: but if we walk in the light as he is in the light , we have fellowship with one another , and the blood of jesus christ his son , cleanses us from all sin .\n",
      "1JN 1:8: if we say that we have no sin , we deceive ourselves , and the truth is not in us .\n",
      "1JN 1:9: if we confess our sins , he is faithful and righteous to forgive us the sins and to cleanse us from all unrighteousness .\n",
      "1JN 1:10: if we say that we haven’t sinned , we make him a liar , and his word is not in us .\n"
     ]
    }
   ],
   "source": [
    "using SIL.Machine.Tokenization;\n",
    "\n",
    "foreach (var row in corpus.Tokenize<LatinWordTokenizer>().Lowercase().NfcNormalize().Take(10))\n",
    "    WriteLine($\"{row.Ref}: {row.Text}\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Corpus processing operations are also available on parallel corpora."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1JN 1:1\n",
      "Source: esta carta trata sobre la palabra de vida que existía desde el principio , que hemos escuchado , que hemos visto con nuestros propios ojos y le hemos contemplado , y que hemos tocado con nuestras manos .\n",
      "Target: that which was from the beginning , that which we have heard , that which we have seen with our eyes , that which we saw , and our hands touched , concerning the word of life\n",
      "1JN 1:2\n",
      "Source: esta vida nos fue revelada . la vimos y damos testimonio de ella . estamos hablándoles de aquél que es la vida eterna , que estaba con el padre , y que nos fue revelado .\n",
      "Target: ( and the life was revealed , and we have seen , and testify , and declare to you the life , the eternal life , which was with the father , and was revealed to us ) ;\n",
      "1JN 1:3\n",
      "Source: los que hemos visto y oído eso mismo les contamos , para que también puedan participar de esta amistad junto a nosotros . esta amistad con el padre y su hijo jesucristo .\n",
      "Target: that which we have seen and heard we declare to you , that you also may have fellowship with us . yes , and our fellowship is with the father and with his son , jesus christ .\n",
      "1JN 1:4\n",
      "Source: escribimos para decirles esto , a fin de que nuestra felicidad sea completa .\n",
      "Target: and we write these things to you , that our joy may be fulfilled .\n",
      "1JN 1:5\n",
      "Source: este es el mensaje que recibimos de él y que nosotros les declaramos a ustedes : dios es luz , y no hay ningún vestigio de oscuridad en él .\n",
      "Target: this is the message which we have heard from him and announce to you , that god is light , and in him is no darkness at all .\n"
     ]
    }
   ],
   "source": [
    "var sourceCorpus = new ParatextTextCorpus(\"data/VBL-PT\");\n",
    "var targetCorpus = new ParatextTextCorpus(\"data/WEB-PT\");\n",
    "var parallelCorpus = sourceCorpus.AlignRows(targetCorpus);\n",
    "\n",
    "foreach (var row in parallelCorpus.Tokenize<LatinWordTokenizer>().Lowercase().NfcNormalize().Take(5))\n",
    "{\n",
    "    WriteLine($\"{row.Ref}\");\n",
    "    WriteLine($\"Source: {row.SourceText}\");\n",
    "    WriteLine($\"Target: {row.TargetText}\");\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "C#"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}