Error with code-blocks translation

[Llith0]

Describe the problem

We are actually trying to translate a md documentation from French to English, there is no problem for the majority of the markdown, but when it come to code-blocks, if a translation is enter, the resulting code-block in the translated website will always end up being "::"

This is the code-block in the French documentation (in the markdown file) :

How it appear on the French website :

But when i'm trying to translate it in English, it always end up being translated as "::" :

I've tried to do it with both code-block and eval_rst, and both end up displaying "::".

I've tried the same process without using myst and using rst parser from sphinx with a code-block, and i was able to translate it without a problem.

I'm sorry i can't provide a link, but it's just a brand new documentation with myst_parser and a code-block.

Link to your repository or website

No response

Steps to reproduce

• Start a new Sphinx project
• Add myst_parser to conf.py
• add the following line : "gettext_additional_targets = ["literal-block"]" in conf.py to generate code-blocks in translation
• add locale_dirs = ['locale/'] in conf.py
• add gettext_compact = False in conf.py
• add a markdown file with a code-block to translate
• generate gettext files : sphinx-build -b gettext source/ potfiles/
• generate po files : sphinx-intl update -p potfiles/ -l en
• Translate code-block
• generate English documentation : sphinx-build -b html source/ build_en/ -Dlanguage=en

The version of Python you're using

Python 3.9.0

Your operating system

Windows 10 build 19042.1052

Versions of your packages

Sphinx 4.2.0
myst-parser 0.15.2
docutils 0.16
sphinx-intl 2.0.1

Additional context

There is not much context, it's just a fresh documentation with one page, myst_parser, and a code-block, then translating it using the translation system in sphinx, and it always end up as "::".

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[chrisjsewell]

Any thoughts @jpmckinney, as I know you did a lot of work on translations?

[jpmckinney]

Huh, I wasn’t aware of the gettext_additional_targets option, so I only added support for the targets that are translatable by default. I’ll try to have a look.

[jpmckinney]

I'm able to reproduce.

[jpmckinney]

Right, so this is the same issue as sphinx-doc/sphinx#8852 For literal-block, Sphinx's Locale transform turns the text to translate into:

::

   text to translate

When MyST Parser then receives this, it renders it as (in doctree):

<raw format="html" translatable="True" xml:space="preserve">
    ::

So, there's the extra :: that the transform added, and MyST-Parser apparently ignores the rest (which isn't where to fix the behavior, since MyST-Parser should not be receiving this RST-specific text to begin with).

Anyway, as mentioned in that issue, Sphinx needs "to develop the mechanism to switch a translation-transform for each parser".

[jpmckinney]

There are also the following options for gettext_additional_targets:

• index: I don't know how to test this. Sphinx's own tests (test_additional_targets_should_be_translated in test_intl.py) don't seem to have a case for it.
• doctest-block: To my knowledge, nothing in Markdown will get parsed as doctest-block. Instead, it's parsed as literal-block, which causes the same issue as reported.
• raw: Same issue as reported for literal-block.
• image: RST image and figure image URLs get rendered untranslated (though the gettext builder does extract the text). However, I guess most users would use Markdown images and figure-md instead.

[jpmckinney]

I've added some more tests in #459. I haven't added a test for rendering HTML with gettext_additional_targets, because the current behavior is incorrect. Below is a test for that once fixed.

FWIW, it seems no one ever tried gettext_additional_targets with recommonmark, but I figure it will have run into the same issue with Sphinx.

diff --git a/tests/test_sphinx/test_sphinx_builds.py b/tests/test_sphinx/test_sphinx_builds.py
index 701aa2e..833cb38 100644
--- a/tests/test_sphinx/test_sphinx_builds.py
+++ b/tests/test_sphinx/test_sphinx_builds.py
@@ -449,6 +449,49 @@ def test_gettext_additional_targets(
     file_regression.check(output, extension=".pot")
 
 
+@pytest.mark.sphinx(
+    buildername="html",
+    srcdir=os.path.join(SOURCE_DIR, "gettext"),
+    freshenv=True,
+    confoverrides={
+        "language": "fr",
+        "gettext_compact": False,
+        "locale_dirs": ["."],
+        "gettext_additional_targets": [
+            "index",
+            "literal-block",
+            "doctest-block",
+            "raw",
+            "image",
+        ],
+    },
+)
+def test_gettext_additional_targets_html(
+    app,
+    status,
+    warning,
+    get_sphinx_app_doctree,
+    get_sphinx_app_output,
+    remove_sphinx_builds,
+):
+    """Test gettext message extraction."""
+    app.build()
+    assert "build succeeded" in status.getvalue()  # Build succeeded
+    warnings = warning.getvalue().strip()
+    assert warnings == ""
+
+    try:
+        get_sphinx_app_doctree(app, docname="index", regress=True)
+    finally:
+        get_sphinx_app_doctree(app, docname="index", resolve=True, regress=True)
+    get_sphinx_app_output(
+        app,
+        filename="index.html",
+        regress_html=True,
+        regress_ext=f".sphinx{sphinx.version_info[0]}.html",
+    )
+
+
 @pytest.mark.sphinx(
     buildername="html", srcdir=os.path.join(SOURCE_DIR, "mathjax"), freshenv=True
 )
diff --git a/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.resolved.xml b/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.resolved.xml
new file mode 100644
index 0000000..c43fa45
--- /dev/null
+++ b/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.resolved.xml
@@ -0,0 +1,90 @@
+<document source="index.md">
+    <section classes="tex2jax_ignore mathjax_ignore" ids="bold-text-1" names="bold\ text\ 1 texte\ 1\ en\ gras">
+        <title>
+            texte 1 en 
+            <strong>
+                gras
+        <paragraph>
+            texte 2 en 
+            <strong>
+                gras
+        <block_quote>
+            <paragraph>
+                texte 3 en 
+                <strong>
+                    gras
+        <note>
+            <paragraph>
+                texte 4 en 
+                <strong>
+                    gras
+        <bullet_list>
+            <list_item>
+                <paragraph>
+                    texte 5 en 
+                    <strong>
+                        gras
+        <enumerated_list enumtype="arabic" suffix=".">
+            <list_item>
+                <paragraph>
+                    texte 6 en 
+                    <strong>
+                        gras
+        <definition_list classes="simple myst">
+            <definition_list_item>
+                <term>
+                    texte 7 en 
+                    <strong>
+                        gras
+                <definition>
+                    <paragraph>
+                        texte 8 en 
+                        <strong>
+                            gras
+        <table classes="colwidths-auto">
+            <tgroup cols="1">
+                <colspec colwidth="100.0">
+                <thead>
+                    <row>
+                        <entry>
+                            <paragraph>
+                                texte 9 en 
+                                <strong>
+                                    gras
+                <tbody>
+                    <row>
+                        <entry>
+                            <paragraph>
+                                texte 10 en 
+                                <strong>
+                                    gras
+        <raw format="html" translatable="True" xml:space="preserve">
+            ::
+        <paragraph>
+            texte 11 en 
+            <strong>
+                gras
+        <paragraph>
+            « 
+            <literal>
+                Backtick
+             » supplémentaire
+        <raw format="html" translatable="True" xml:space="preserve">
+            ::
+        <literal_block language="none" linenos="False" translatable="True" xml:space="preserve">
+            ::
+        <literal_block language="default" linenos="False" translatable="True" xml:space="preserve">
+            ::
+        <literal_block language="json" linenos="False" translatable="True" xml:space="preserve">
+            ::
+        <raw format="html" translatable="True" xml:space="preserve">
+            ::
+        <literal_block language="python" linenos="False" translatable="True" xml:space="preserve">
+            ::
+        <raw format="html" translatable="True" xml:space="preserve">
+            ::
+        <paragraph>
+            <image alt="Poisson amusant 1" candidates="{'*': 'poisson-amusant.png'}" uri="poisson-amusant.png">
+        <image alt="Fun Fish 2" candidates="{'*': 'fun-fish.png'}" translatable="True" uri="fun-fish.png">
+        <figure>
+            <image alt="Fun Fish 3" candidates="{'*': 'fun-fish.png'}" translatable="True" uri="fun-fish.png">
diff --git a/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.sphinx4.html b/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.sphinx4.html
new file mode 100644
index 0000000..8ad6a50
--- /dev/null
+++ b/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.sphinx4.html
@@ -0,0 +1,156 @@
+<div class="documentwrapper">
+ <div class="bodywrapper">
+  <div class="body" role="main">
+   <section class="tex2jax_ignore mathjax_ignore" id="bold-text-1">
+    <h1>
+     texte 1 en
+     <strong>
+      gras
+     </strong>
+     <a class="headerlink" href="#bold-text-1" title="Lien permanent vers ce titre">
+      ¶
+     </a>
+    </h1>
+    <p>
+     texte 2 en
+     <strong>
+      gras
+     </strong>
+    </p>
+    <blockquote>
+     <div>
+      <p>
+       texte 3 en
+       <strong>
+        gras
+       </strong>
+      </p>
+     </div>
+    </blockquote>
+    <div class="admonition note">
+     <p class="admonition-title">
+      Note
+     </p>
+     <p>
+      texte 4 en
+      <strong>
+       gras
+      </strong>
+     </p>
+    </div>
+    <ul class="simple">
+     <li>
+      <p>
+       texte 5 en
+       <strong>
+        gras
+       </strong>
+      </p>
+     </li>
+    </ul>
+    <ol class="arabic simple">
+     <li>
+      <p>
+       texte 6 en
+       <strong>
+        gras
+       </strong>
+      </p>
+     </li>
+    </ol>
+    <dl class="simple myst">
+     <dt>
+      texte 7 en
+      <strong>
+       gras
+      </strong>
+     </dt>
+     <dd>
+      <p>
+       texte 8 en
+       <strong>
+        gras
+       </strong>
+      </p>
+     </dd>
+    </dl>
+    <table class="colwidths-auto docutils align-default">
+     <thead>
+      <tr class="row-odd">
+       <th class="head">
+        <p>
+         texte 9 en
+         <strong>
+          gras
+         </strong>
+        </p>
+       </th>
+      </tr>
+     </thead>
+     <tbody>
+      <tr class="row-even">
+       <td>
+        <p>
+         texte 10 en
+         <strong>
+          gras
+         </strong>
+        </p>
+       </td>
+      </tr>
+     </tbody>
+    </table>
+    ::
+    <p>
+     texte 11 en
+     <strong>
+      gras
+     </strong>
+    </p>
+    <p>
+     «
+     <code class="docutils literal notranslate">
+      <span class="pre">
+       Backtick
+      </span>
+     </code>
+     » supplémentaire
+    </p>
+    ::
+    <div class="highlight-none notranslate">
+     <div class="highlight">
+      <pre><span></span>::
+</pre>
+     </div>
+    </div>
+    <div class="highlight-default notranslate">
+     <div class="highlight">
+      <pre><span></span><span class="p">::</span>
+</pre>
+     </div>
+    </div>
+    <div class="highlight-json notranslate">
+     <div class="highlight">
+      <pre><span></span><span class="p">::</span>
+</pre>
+     </div>
+    </div>
+    ::
+    <div class="highlight-python notranslate">
+     <div class="highlight">
+      <pre><span></span><span class="p">::</span>
+</pre>
+     </div>
+    </div>
+    ::
+    <p>
+     <img alt="Poisson amusant 1" src="_images/poisson-amusant.png"/>
+    </p>
+    <img alt="Fun Fish 2" src="_images/fun-fish.png"/>
+    <figure class="align-default">
+     <img alt="Fun Fish 3" src="_images/fun-fish.png"/>
+    </figure>
+   </section>
+  </div>
+ </div>
+</div>
diff --git a/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.xml b/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.xml
new file mode 100644
index 0000000..22524ea
--- /dev/null
+++ b/tests/test_sphinx/test_sphinx_builds/test_gettext_additional_targets_html.xml
@@ -0,0 +1,90 @@
+<document source="index.md">
+    <section classes="tex2jax_ignore mathjax_ignore" ids="bold-text-1" names="bold\ text\ 1 texte\ 1\ en\ gras">
+        <title>
+            texte 1 en 
+            <strong>
+                gras
+        <paragraph>
+            texte 2 en 
+            <strong>
+                gras
+        <block_quote>
+            <paragraph>
+                texte 3 en 
+                <strong>
+                    gras
+        <note>
+            <paragraph>
+                texte 4 en 
+                <strong>
+                    gras
+        <bullet_list>
+            <list_item>
+                <paragraph>
+                    texte 5 en 
+                    <strong>
+                        gras
+        <enumerated_list enumtype="arabic" suffix=".">
+            <list_item>
+                <paragraph>
+                    texte 6 en 
+                    <strong>
+                        gras
+        <definition_list classes="simple myst">
+            <definition_list_item>
+                <term>
+                    texte 7 en 
+                    <strong>
+                        gras
+                <definition>
+                    <paragraph>
+                        texte 8 en 
+                        <strong>
+                            gras
+        <table classes="colwidths-auto">
+            <tgroup cols="1">
+                <colspec colwidth="100.0">
+                <thead>
+                    <row>
+                        <entry>
+                            <paragraph>
+                                texte 9 en 
+                                <strong>
+                                    gras
+                <tbody>
+                    <row>
+                        <entry>
+                            <paragraph>
+                                texte 10 en 
+                                <strong>
+                                    gras
+        <raw format="html" translatable="True" xml:space="preserve">
+            ::
+        <paragraph>
+            texte 11 en 
+            <strong>
+                gras
+        <paragraph>
+            « 
+            <literal>
+                Backtick
+             » supplémentaire
+        <raw format="html" translatable="True" xml:space="preserve">
+            ::
+        <literal_block language="none" translatable="True" xml:space="preserve">
+            ::
+        <literal_block language="default" translatable="True" xml:space="preserve">
+            ::
+        <literal_block language="json" translatable="True" xml:space="preserve">
+            ::
+        <raw format="html" translatable="True" xml:space="preserve">
+            ::
+        <literal_block language="python" translatable="True" xml:space="preserve">
+            ::
+        <raw format="html" translatable="True" xml:space="preserve">
+            ::
+        <paragraph>
+            <image alt="Poisson amusant 1" candidates="{'*': 'poisson-amusant.png'}" uri="poisson-amusant.png">
+        <image alt="Fun Fish 2" candidates="{'*': 'fun-fish.png'}" translatable="True" uri="fun-fish.png">
+        <figure>
+            <image alt="Fun Fish 3" candidates="{'*': 'fun-fish.png'}" translatable="True" uri="fun-fish.png">

[jeanas]

In case that helps anyone, I was able to work around this issue by putting this deliciously awful workaround in an extension:

from sphinx.transforms import i18n

class ModifiedIndent:
    def __init__(self, s, _):
        self.s = s
    def __radd__(self, _):
        return f"```\n{self.s}\n```"

i18n.indent = ModifiedIndent