Dynamic Examples

Usually ALAN books and tutorials will contain code and transcripts examples, used to illustrate how to implement a specific task and providing a sample of its in-game output.

Manually editing similar code and transcript examples is not a reliable method, because its error prone. Copying and pasting excerpts from real sample adventures and their transcripts is more reliable, but only in the short term — as times goes by, the code example might stop working if it depends on a library or extension which was updated, and the generated transcript might no longer match the actual in-game text, for the same reasons or because of internal changes to the ALAN language itself.

So, what's the solution to this problem?

Meet "Dynamic Examples and Transcripts", a novel method to ensure that code and transcript in ALAN documents always mirror the latest state of their library dependencies and the ALAN language.

With this approach, even if the examples or Library code were changed, these changes will always be reflected into the documentation, without manual efforts. This is definitely worth the effort of setting up a slightly more complex toolchain.

Dynamic Examples and Transcripts

When we decided to port the StdLib documentation from PDF to AsciiDoc, we realized that many examples were residual from older versions of the library and no longer worked with its latest version. So we decided to adopt a novel approach to the problem by exploiting Asciidoctor include:: directive to extract the required code and transcripts selections dynamically, from real source examples and their auto-generated transcripts, to ensure that every time we rebuild the documentation these snippets would mirror the current state of the library.

We named this technique "Dynamic Examples and Transcripts".

The benefits of adopting this approach are twofold:

1. Documentation code examples and transcripts will always mirror their real world usage, even if the examples sources were updated, or the library or extension they depend on were updated in the meantime, since the transcripts are regenerated from a freshly compiled adventure each time.

   This "set it and forget" approach frees documentation maintainers from the burden of having to manually updated documentation examples and library excerpts, separating concerns between code and documentation entertainment (especially regarding external libraries which might be maintained by someone else than the documentation curator).

2. If a code example ceases to work due to changes in its library or extension dependencies, or due to changes in the ALAN language, the documentation toolchain will abort due to the ALAN compiler error, bringing the issues immediately to the attention of the document maintainer, who'll be able to solve the problem by updating the code examples accordingly, or contact the maintainer of the library or extension about the breaking changes.

This article will describe how this technique works, providing practical examples of its usage and implementation details. We strongly advise all ALAN documentation maintainers to adopt this method in their AsciiDoc workflow.

Selective Contents Inclusion

Selectively including certain parts of a source file can be done in two ways in Asciidoctor:

1. By providing line ranges.
2. Via tagged regions.

Including by Line Ranges

The first solution doesn't require changing the sources in any way, it's enough to provide the desired line range to import from the target source:

include::lib_verbs.i[lines="1..10,15..20"]

but its obvious downside is that whenever the target source changes the line ranges might need to be adjusted, because the desired code snippet might have shifted up or down in the source.

Including by Tagged Regions

The second solution is much better because it allows to tag specific regions of interest in the target sources using their native comments, which can than be imported via their tag name.

This system is already being used to handle dynamically the code examples in the StdLib Manual, in order to include the examples snippets from real sources, which are then compiled and run with a commands script, and then snippets of the actual game-play are included in the Manual via the same system.

Here's a real example from _variations1.alan:

--------------------------------------------------------------------------------
-- §6.1.3. DOOR
--------------------------------------------------------------------------------
-- tag::front_door[]
THE front_door ISA DOOR AT garden
  NAME front door
  DESCRIPTION ""
END THE front_door.
-- end::front_door[]

where the comment lines -- tag::front_door[] and -- end::front_door[] delimit the front_door tagged region, which is then imported into the manual (in StdLibMan_06.adoc) via:

[source,alan]
--------------------------------------------------------------------------------
include::{utf8dir}/_variations1.alan[tag=front_door]
--------------------------------------------------------------------------------

In the final Manual, the reader will just see a normal ALAN code block, but for us maintainers it means that whenever we change the example sources, or their output changes due library changes, the Manual is self-updated without need of any cut-&-paste operations. It's a slightly more laborious approach, but it's worth it because it only needs to be done once (set it and forget it).

NOTE — Asciidoctor will automatically strip off region tags when including contents via include::, so you don't have to worry about nested or overlapping tagged-regions — any comments matching a region tag will not show up in the final document.

Being able to use tagged regions comments in Alan sources and transcripts facilitates selectively including parts of the code and transcripts, without having to worry about if line positions shift around in them.

The obvious downside of this approach is that it requires adding the region-tags comments in the StdLib sources. Whereas these comments are not a problem in the examples files (which either are not shared with end users, or if they are they get stripped of these comments in the release archive), adding these comments to the library sources is another issue altogether.

Our solution regarding the presence of tag-region comments in the library sources was to filter them out in the final end-user release. This adds an extra layer of complexity, because the library sources need to be preprocessed for the end users (using sed), but it has already been adopted in the Alan-Docs repository, for the The Alan Beginner's Guide (which extensively imports snippets from the example adventure, which is then made available to end users after stripping the tag-comments line).

Toolchain Details

Here's a description of how the StdLib toolchain for generating its Manual and other documentation works.

The update.sh script carries out multiple tasks. For every source documentation folder <foldername>:

1. Compile Adventures — Compile every adventure inside the source folder.
2. Generate Transcripts — Run every compiled adventure against one or more .a3s solution files whose name contains the adventure name (i.e. <adv-name>*.a3s) and save the transcript as a sane named .a3t file.
3. Format Transcipts to AsciiDoc — Run every .a3t file generated by the previous step through a SED script that will convert the verbatim transcript into a well-formatted AsciiDoc example block:
   • Convert special characters (that could be interpreted as formatting) into its predefined Asciidoctor attribute for replacement equivalent (or its HTML entity equivalent, if no attribute is available).
   • Preserve hard line-breaks by adding + at end-of-line, where required.
   • Style player input in emphasis.
   • Style comments in player input (;) via #[comment]..#.
   • Hide region tags in player input by converting the line into an AsciiDoc comment //. This is needed because region-tag comments are defined in the solution file, which means that in the transcript these comments will appear after the prompt, so Asciidoctor won't be able to skip them when using include:: directives. Since transcripts are formatted as example blocks in the ADoc source, comment lines will be ignored (unlike in code blocks).
4. Build HTML Docs — Convert every *.asciidoc document inside the source folder into a standalone HTML file in the destination folder ../<destination path>/<foldername>/.
5. Sanitize Alan Sources — Take every Alan source adventure whose name doesn't start with underscore (i.e. $srcDir/[^_]*.alan), strip away all AsciiDoc region-tag comment lines, and copy it to the destination folder. In other words, underscored adventures are for internal documentation use only, while the others are (also) real examples for end users.

SED Script for Transcripts Sanitation

As a practical example, we've included in the Wiki the actual SED script used by the StdLib project:

• sanitize_a3log.sed

The scripts carries out both essential and aesthetic tasks, the former being required for proper AsciiDoc inclusion, the latter to provide better formatted transcripts in the final documentation (requires dedicated styles definitions in the backend template, e.g. CSS stylesheets for HTML).

Here's the contents of the script:

# "sanitize_a3log.sed"                  v0.0.3 | 2021/10/20 | by Tristano Ajmone
# ******************************************************************************
# *                                                                            *
# *                         Sanitize Alan Transcripts                          *
# *                                                                            *
# ******************************************************************************
# This sed script will reformat a verbatim Alan transcript into a well-formed
# AsciiDoc example block, which can then be included into a document.
# ------------------------------------------------------------------------------
# exclude from processing player input lines with region tags:
/^>\s*;\s*(tag|end)::/ !{
    # ===================
    # Chars Substitutions
    # ===================
    s/\*/{asterisk}/g     #   *   ->   {asterisk}
    s/\\/{backslash}/g    #   \   ->   {backslash}
    s/\x60/{backtick}/g   #   `   ->   {backtick}
    s/\^/{caret}/g        #   ^   ->   {caret}
    s/\+/{plus}/g         #   +   ->   {plus}
    s/~/{tilde}/g         #   ~   ->   {tilde}
    s/\[/{startsb}/g      #   [   ->   {startsb}
    s/]/{endsb}/g         #   ]   ->   {endsb}
    s/\_/\_/g      #   _   ->   _
    s/#/\#/g          #   #   ->   #
    # ==========================
    # Process player input lines
    # ==========================
    /^>/ {
        # ignore empty input lines:
        /^>\s*$/ !{
            # ignore input with only comments:
            /^>\s*;/ !{
                # ----------------------
                # Italicize Player Input
                # ----------------------
                # Unconstrained Italics
                s/^(>\s+)([^[:alpha:][:digit:]\s][^;]*?|[^;]*?[^[:alpha:][:digit:]\s])(\s*?)( ;.*)?$/\1__\2__\3\4/
                # Constrained Italics
                s/^(> *?)(\w[^;]*?\w)(\s*)(;.*)?$/\1_\2_\3\4/
            }
            # ---------------------
            # Player Input Comments
            # ---------------------
            s/(>[^;]+)(;.*)$/\1[.comment]#\2#/
        }
        # -----------------
        # fix '>' to '>'
        # -----------------
        s/^>/\>/
    }
    # =========================
    # Preserve hard line breaks
    # =========================
    # ignore empty lines:
    /^$/ !{
        s/$/ +/
    }
}
# ====================
# AsciiDoc Region Tags
# ====================
# Convert to AsciiDoc commented line, so it won't be shown in document:
s/^>\s*;\s*((tag|end)::.*)$/\/\/ \1/

Sanitizing Sources in Releases

While region-tag comments are useful in examples and library sources, to allow selective inclusion of specific snippets, keeping them in the distributed sources would make them look ugly and confusing, so it's useful to sanitize the sources before releasing them.

Stripping region-tags comments from library sources and code examples before including them in release packages boils down to processing the whole file, line by line, and removing (or skipping during line-by-line copy) any line that begins with an ALAN comment-line delimiter followed by tag:: or end::.

This can be easily achieved via SED.

TBD! — This section still needs to be completed.

---

References

• AsciiDoc Language Documentation:
  • Include Directive
    • Tagging regions
    • Include Content by Line Ranges