Smart Typography and Its Challenges

[tajmone]

Just wanted to share a few thought on smart typography, auto-replacements, and some other features commonly found in lightweight markup languages, and how these could impact PML's future roadmap.

Many (if not most) lightweight syntaxes support smart typography to some degree. The most common examples are smart quotes (single and double), em dashes, ellipsis, etc.

For example, Pandoc has the typography extension, which is auto-enabled on many formats, and can be disabled if needed:

Extension: smart

Interpret straight quotes as curly quotes, --- as em-dashes, -- as en-dashes, and ... as ellipses. Nonbreaking spaces are inserted after certain abbreviations, such as “Mr.”

whereas in Asciidoctor similar functionality is natively enabled, although its differently supported, since quotes need to be back-ticked to be curly (e.g. "`quoted`" and '`quoted`") which was done in order to allow straight quotes, since AsciiDoc is often used for software documentation. But Asciidoctor natively handles apostrophe syntax:

When entered with the ' key, Asciidoctor renders an apostrophe that is directly preceded and followed by a character, such as in contractions and possessive singular forms, as a curved apostrophe. This inconsistent handling of apostrophes and quotation marks is a hold over from the original AsciiDoc processor. An apostrophe directly bounded by two characters is processed during the replacements substitution phase, not the inline formatting (quotes) phase. This is why an apostrophe directly followed by white space, such as the possessive plural form, is not curved by default.

As the above paragraph illustrates, the char-bounded apostrophe is being handled based on the assumption that the document is in English — when it comes to smart typography, locale-specific features and assumption can easily turn into double-edged swords. Also, there's the inconsistent usage problem, which stems from legacy, but consistency would have been better in this respect.

When it comes to dashes, AsciiDoctor will treat two hyphens (--) as an em-dash (—) — but only if surrounded by spaces! — which was a huge mistake (again, probably from the AsciiDoc legacy) since it hinders creating en-dashes; whereas Pandoc got this right from the onset. In academic papers, and in good book publishing, it's simply unacceptable to use an hyphen to separate number ranges (e.g. 1990-93), whereas an en-dash needs to be used (e.g. 1990–93). For Asciidoctor the solution came as a dedicated third party extension to fix this. But also the condition of the em-dash being surrounded by spaces is questionable, since the use of this typographical element is subject to different rules in different contexts (academia vs novels) and in different languages.

All this is to say that while features like smart typography and auto-replacements are highly demanded and practical, one needs to consider carefully all the possible context in which the syntax will be used (academic papers, programming documentation, novels, etc.) as well as the differnt languages it will be used with.

For example, double quotes are used differently in some languages, notably the German language which uses an upside-down opening mark („) instead:

https://www.thoughtco.com/german-zeichensetzung-punctuation-marks-4082218

The French language has some strict rules regarding the insertion of thin-spaces between the surrounding quotes and the quoted word/sentence; and some languages (French, Italian, and others) still use double angled-brackets (« » aka "corporal/chevron bracket") as an alternative to double quotes.

All of these factors should be kept into account when considering the possibility of adding to PML smart typography and substitutions in the future. E.g. PML 1.3 replaced parameters enclosures from [[ ]] to << >>, which might affect future implementation of double angled-brackets substitutions — e.g. <<quoted>> → «quoted» — which might be important in many locales.

Although it's indeed possible to simply insert the required characters directly in the source, without having to rely on substitutions (and most of these punctuation symbols are part of Latin1/ISO-8859-1), the main reason why lightweight markup prefers sticking to ASCII/ISO symbols is due to monospace fonts often not covering those glyphs, and CLI/version control tools also relying on monospace fonts (in other words, using directly glyphs like “ ” ‘ ’ « » – — in the source files might lead to unreadable diffs in Git, or misrepresented characters in the terminal).

If (and to which degree) a syntax should natively support such features largely depends on its target audience: is it a syntax targeting academic writers, programmers, or novelists? Which languages and alphabets should be supported? Or should the core syntax be usage- and locale-agnostic, and provide all these features by third party (or extra) packages which can leverage the syntax API?

From my experience, it would be quite hard to create a "universal syntax" that could work just as well with any language and alphabet (including RTL languages like Arabic, Hebrew, Farsi, etc.) and that could be versatile enough to be used in any writing context (from novels to software documentation). Such a syntax would need to offer a reduced set of native features, and delegate to external packages all the advanced features. In theory, it's doable, but it would require a lot of effort.

It's often a better choice to limit the scope of a syntax in view of its target audience. E.g. by settling to support only Latin alphabets (in terms of features, not characters support) and targeting a specific type of documentation. Computer documentation is a different beast from narrative, they have different needs and standards. Single article publications and books (or multi-part tutorials) also have different requirements (e.g. cross-document X-Refs, footnotes and end-notes numeration spanning across multiple docs, etc.).

For these reasons, I though it would be a good idea to open this Discussion, by starting it from smart typography — which, IMO, is a feature around which the different needs tend to emerge and clash very soon.

Hopefully, we'll see more feedback from end users on this topic, which will allow PML to be shaped around its userbase needs.

---

References

• Asciidoctor Manual » Character Replacement Substitutions
• Asciidoctor Manual » Special Character Substitutions
• Asciidoctor Manual » Quotation Marks and Apostrophes

[pml-lang]

Excellent summary of the usefulness and challenges of smart typography. Thank you.

According to your explanations it seems to be impossible to provide one-size-fits-all features. Therefore I suggest to make auto-replacements totally configurable (not hard-coded), and provide the following features in PML:

• Integrated basic support for commonly used auto-replacements that have easy rules (no exceptions) and work well in all cases. Enabled by default, easy to disable.
• User-friendly support for writers to define their own auto-replacement rules, in order to support locale-specific and domain-specific features.

To achieve this, auto-replacement rules should be stored in one or more configuration files that can easily be enabled/disabled/modified by users. Thus, users from France, for example, can define their country-specific rules in a separated config file, and share it with other people who write documents in French.

PML 1.3 replaced parameters enclosures from [[ ]] to << >>

Yes, and that was a bad idea. <<constant_name>> will probably be replaced by [= constant_name] in a future version.

[coemgenuslcp]

I am all in favor of this concept, not only for the sake of locales and specific cases, but also in rebellion against norms that I can't stand. For example:

• USA closes quotations with commas inside the close quote mark, not outside, only because it looks better, even though the comma is not always part of the quotation.
• Word processing, internet, and now texting have decimated any custom of the spaced ellipsis (. . .), as well as the fullstop ellipsis (. . . ., ellipsis with a fullstop point before it), auto-replacing any three points with the … character. Worse, Chicago Manual of Style these days is favoring ellipses without spaces before and after them. Same with em dashes.

[tajmone]

i18n Quotation Marks and Formatting

The following link contains a useful table summarizing how Asciidoctor quotation marks are rendered in different locales by an alternative HTML renderer:

• https://github.com/jirutka/asciidoctor-html5s/blob/master/README.adoc#quotes

It also lists various typographical replacements, so the document is a valuable reference when it comes to advanced typography in lightweight markup languages and internationalization needs.