Implement UAX31's quick check algorithm

dlang · Apr 20, 2024 · fbc02c1 · fbc02c1
1 parent e397d5e
commit fbc02c1
Show file tree

Hide file tree

Showing 23 changed files with 2,667 additions and 4,377 deletions.
diff --git a/changelog/dmd.identifier-quick-check.dd b/changelog/dmd.identifier-quick-check.dd
@@ -0,0 +1,14 @@
+Identifiers are verifiable against normalization form C
+
+It is now possible to verify identifiers against normalization form C, as per UAX31 and C23.
+
+The quick check algorithm that is in use is an incomplete form that does not handle maybes and can result in false positives in errors.
+The purpose of this algorithm in its incomplete state is to minimize normalization in a future update.
+
+It is configurable for both D and ImportC using ``-normalization=<strategy>`` and ``-normalization-importc=<strategy>``.
+Using the options ``ignore``, ``deprecate``, ``warn``, and ``error``.
+
+Unless you use non-ASCII characters, this change will not affect you.
+
+Due to bug [24507](https://issues.dlang.org/show_bug.cgi?id=24507) if you use GCC as your preprocessor with any identifier that is not normalized to form C, by default it will produce a warning.
+Use the GCC option ``-Wnormalized=none`` to disable this [warning](https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wnormalized_003d), and the D compiler will take over.
diff --git a/compiler/src/dmd/cli.d b/compiler/src/dmd/cli.d
@@ -639,6 +639,24 @@ dmd -cov -unittest myprog.d
             `Turns off all array bounds checking, even for safe functions. $(RED Deprecated
             (use $(TT $(SWLINK -boundscheck)=off) instead).)`,
         ),
+        Option("normalization=<strategy>",
+            "Specify what to do when an identifier is not normalized",
+            `The strategy that will be used when an identifier may not be normalized (quick check algorithm).
+            The quick check algorithm will give off false positives as it does not handle the maybe case.
+                $(UL
+                    $(LI $(I ignore): Silently accept any failing identifiers (default))
+                    $(LI $(I warn): Emit a warning but accept)
+                )`
+        ),
+        Option("normalization-importc=<strategy>",
+            "Specify what to do when an identifier is not normalized",
+            `The strategy that will be used with ImportC when an identifier may not be normalized (quick check algorithm).
+            The quick check algorithm will give off false positives as it does not handle the maybe case.
+                $(UL
+                    $(LI $(I ignore): Silently accept any failing identifiers (default))
+                    $(LI $(I warn): Emit a warning but accept)
+                )`
+        ),
         Option("nothrow",
             "assume no Exceptions will be thrown",
             `Turns off generation of exception stack unwinding code, enables