epic: conversion of Keyman Engine for Web to ES modules 🧩

[jahorton]

This is the base branch for an ongoing conversion of Keyman Engine for Web code; when complete, it will be based upon use of ES modules rather than the old "namespace"-style structure it was previously using.

Other goals include complete modularization of Keyman Engine for Web's on-screen keyboard component. While still included in KMW, it will also be able to function independently and be compilable as a standalone project. As we are considering reuse of Web's OSK within the desktop-platform Keyman apps, this will be a very notable step toward accomplishing such a goal.

At this time, this branch also includes an update for Web's README.md corresponding to the changeset from the 📜 set of Web code reorganization PRs ending with #7831. (It should be re-updated again near completion.)

---

Approximate state of modularization so far (so GitHub won't hide the comment on us):

graph TD;
    OSK[web/src/engine/osk];
    KP["common/web/keyboard-processor"];
    IP["common/web/input-processor"];
    OSK-->KP;
    IP-->KP;
    Utils["common/web/utils"];
    KP---->Utils;
    Wordbreakers["common/models/wordbreakers"];
    Models["common/models/templates"];
    Models-->Utils;
    LMWorker["common/web/lm-worker"];
    LMWorker-->Models;
    LMWorker-->Wordbreakers;
    LMLayer["common/predictive-text"];
    LMLayer-->LMWorker;
    IP-->LMLayer;

    subgraph PredText["WebWorker + its interface"]
        LMLayer;
        LMWorker;
        Models;
        Wordbreakers;
    end

    subgraph Headless["Fully headless components"]
        direction LR
        KP;
        IP;
        Utils;
        PredText;
    end

    subgraph ClassicWeb["Previously unmodularized components"]
        Device[web/src/engine/device-detect];
        Device----->Utils;
        Elements[web/src/engine/element-wrappers];
        Elements-->KP;
        KeyboardCache[web/src/engine/package-cache];
        KeyboardCache-->IP;
        DomUtils[web/src/engine/dom-utils];
        DomUtils-->Utils;
        OSK-->DomUtils;
        OSK---->IP;
        Configuration[web/src/engine/paths];
        Configuration-->OSK;
        CommonEngine[web/src/engine/main];
        CommonEngine-->Configuration;
        CommonEngine-->Device;
        CommonEngine-->KeyboardCache;
        CommonEngine-->OSK;
        Attachment[web/src/engine/attachment];
        Attachment-->DomUtils;
        Attachment-->Elements;
    end

    subgraph WebEngine["Keyman Engine for Web (top-level libraries)"]
        Browser[web/src/app/browser];
        WebView[web/src/app/webview];

        WebView--->CommonEngine;

        Browser--->CommonEngine;
        Browser-->Attachment;
    end

Loading

(This technically includes changes from not-yet-published commits, but it's pretty accurate to where things are headed.)

Note that the element-wrappers child package will be referenced by the app/browser target, which is not included in the graph above. (It'd rest higher up.)

---

Pending tasks of note:

• Fully modularize all of KMW

  • Achieved by everything through chore(web): final modularization cleanup 🧩 #8891.

• Unit test scripts for CI builds should be defined in the new ./ci.sh test action from feat(web): implements shell scripting for CI release-build configurations 📜 #8001, rather than Web's ./build.sh or ./test.sh.

  • Handled by fix(web): build config cleanup, Safari & mobile-device bugfixes (thanks to CI checks) 🧩 #8830.

• chore(web): Eliminate module deep-references 🧩 #8148

• Address, then resolve all comments for change(web): element wrapper subpackage modularization 🧩 #8055.

  • They're loose threads that were unresolvable at the time.
  • Being handled by fix(web): restores element text-selection auto-scrolling 🧩 #8963.

• SO MANY loose threads for change(web): spins off the OSK component as its own subproject, converts it to use ES modules 🧩 #8056... but there's no clear checklist for them.

  • General guideline for this: check all doc-comments for all "EventMap" types in the following files:
    • oskView.ts
    • visualKeyboard.ts
    • There may be others, but those are the main two of note... and that I could find with preserved 'original implementation' code.
    • Partially(?) addressed by fix(web): app/browser osk hooks, initialization 🧩 #8630.

• Ensure that no regression in line with bug(web/engine): incorrect CSS classes for keyboards with embedded namespacing #3910 occurs. (Can be expensive long-term if we let that one slip by!)

  

  (Verified with chore(web): final modularization cleanup 🧩 #8891)

• Follow-up on refactor(web): browser focus-maintenance states 🧩 #8624 (comment): duplication of OSK hide & show events

  • See chore(web): deduplicates non-distinct osk events 🧩 #8970.

• There are some notable inconsistencies with internal event nomenclature, as noted by refactor(web): proper modularization, implementation of documented API events 🧩 #8591 (comment)

  • Handled in change(web): internal-event convention consistency 🧩 #9012.

• I've probably mucked up a lot of the tsconfig project references. It's "fine" when using our build script infrastructure, but it'd likely lead to unintended compilation issues if someone were to ignore them.

  • Handled by fix(web): build config cleanup, Safari & mobile-device bugfixes (thanks to CI checks) 🧩 #8830.

• It would be wise to look through the various build-bundler.js implementations and determine a common set of config options that could be inherited by all components.

  • Handled by chore(web): esbuild configuration cleanup 🧩 #9025.

• It would also be wise to look through all the tsconfig.json configurations and determine the maximal common set for Web.

  • We'll want something like this for the full repo eventually. Having a top-level Web variant should help with that future effort.
  • Handled in chore(web): centralizes all KMW & related tsconfig settings 🧩 #9011.

• Test against this keyboard: https://help.keyman.com/keyboard/english_shavian_qwerty/1.0/english_shavian_qwerty

  • A published keyboard that actually implements K_ROPT (the "hide keyboard" key) on the touch layout, even in-app. An edge case I'll easily miss if not careful.
  • Verified with chore(web): final modularization cleanup 🧩 #8891 on Android. Should still add an Android and an iOS user test for it.

• As requested at fix(web): KMW recorder modularization, restoration 🧩 #8821 (comment), we should add explicit documentation for public-API functions and methods.

  • That comment was for a specific instance, but the principle is probably intended for all API functions.
  • Addressed by docs(web): demarcation of methods supporting Web's public API 🧩 #9036.

• Thorough testing of selection, start-of-context behavior, etc.

  • feat(common/web): mocks may now model selected text 🧩 #8462 (comment)
  • See bug(iOS): selecting text and then backspacing also deletes the character before the selection #7865 and bug(web): pressing layer switch key with selected text would delete the selection #7866. We haven't fully addressed them yet, but there's some material in there good for reproducing certain issues that could be adaptable into user tests.

• Re user-testing:

  • https://github.com/keymanapp/keyman/pull/8905/files#r1222280122
    • Developer test-host page testing on mobile devices / under mobile emulation
  • See fix(web): restores element text-selection auto-scrolling 🧩 #8963.

• Before merging, run a full regression-test suite on KMW and the mobile apps in order to find possible regressions.

• Update keymanweb.com to use boolean false instead of string 'false' for KMW's setActiveOnRegister initialization option when using KMW 17.0+: change(web): facilitates differentiated configuration objects for KMW variants 🧩 #8458 (comment)

  • See change: setActiveOnRegister now boolean in KMW 17.0 keymanweb.com#75, which should merge when this lands.

Note: Refer also to #8047, the original version of this PR before a git merge disaster happened on what was intended as a child PR.

User Testing

Due to the scale of user testing involved, the actual user tests will be posted under separate issues:

TEST_MAIN_WEB: See #9037 / #9037 (comment).

• Only pass when all tests in test(web): feature-branch regression testing - ES modules (🧩), main suite #9037 pass acceptably.

TEST_OTHERS: See #9039 / #9039 (comment)

• Only pass when all tests in test(common/web): feature-branch regression testing - ES modules (🧩), embedded / alt uses #9039 pass acceptably.

TBD(?): do the mobile apps need their own full regression tests again?

[keymanapp-test-bot]

User Test Results

Test specification and instructions

• ✅ TEST_MAIN_WEB (PASSED): - Though an issue did arise for one of the tests due to differences between a test keyboard's phone and tablet layouts. I've made tweaks accordingly. (notes)
• ✅ TEST_OTHERS (PASSED)

Test Artifacts

• Android
  • Keyman for Android apk
  • FirstVoices for Android 16.0 apk
  • FirstVoices Keyboards for Android apk
  • KeyboardHarness apk
  • Keyman for Android 16.0 apk
  • KMSample1 apk
  • KMSample2 apk
• Developer
  • Keyman Developer
  • Compiler Regression Tests
  • kmcomp.zip
• iOS
  • Keyman for iOS (simulator image)
  • FirstVoices Keyboards for iOS (simulator image)
  • TestFlight internal PR build version - 17.0.130 (0.8560.8757)
• Keyboards
  • Test Keyboards
• Linux
  • Keyman for Linux
  • Keyman for Linux
• macOS
  • Keyman for macOS
• Web
  • KeymanWeb Test Home
• Windows
  • Keyman for Windows
  • FirstVoices Keyboards for Windows
  • Text Editor (32 bit)
  • Text Editor (64 bit)

[jahorton]

For any changes before this point, refer to #8047 and its descendant PRs, which were all merged properly after reviews - that is, before the accidental merge disaster triggered by merging #8549, which had its base branch improperly set.

Anything after this point is from work after the necessary git-reversion shenanigans that were needed to patch up the state of the repo.

[jahorton]

TEST_OTHERS: PASSED

TEST_MAIN_WEB: PASSED - Though an issue did arise for one of the tests due to differences between a test keyboard's phone and tablet layouts. I've made tweaks accordingly.

The quick version there: khmer_angkor's tablet layout doesn't do subconsonant subkeys like its phone layout. The original version of one test's spec didn't account for that.

• When merging, copy test(web): feature-branch regression testing - ES modules (🧩), main suite #9037's description over as the new content for the Platform Acceptance Test for Web wiki page.
  • But do include the wiki page's SUITE_INSPECT in its original position.
  • Do a find-and-replace on its URL links when copying the description text over: Keymanweb_TestPullRequests/390024:id/src => Keymanweb_TestPullRequests/TBD:id/src, since that's the established pattern for the Wiki's version of acceptance testing.

[mcdurdin]

LGTM

[keyman-server]

Changes in this pull request will be available for download in Keyman version 17.0.130-alpha