Skip to content
This repository has been archived by the owner on Nov 29, 2023. It is now read-only.

Refactor transform to be isomorphic (Node/web compatibility) #171

Merged
merged 24 commits into from
Mar 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
e02efa9
Convert `transform` to async/await
eyelidlessness Feb 18, 2023
6242fbe
Initial Node/DOM compatibility
eyelidlessness Feb 19, 2023
916e2e3
Fix: consistent URL escaping in web environments
eyelidlessness Feb 19, 2023
a0238fe
Fix: HTML document/"fragment" hierarchy
eyelidlessness Feb 19, 2023
cda915a
Fix: namespaces on instance root nodes
eyelidlessness Feb 19, 2023
b46e7b5
Remove fatal errors from XSL from unsupported extensions
eyelidlessness Feb 19, 2023
29a733f
Fix: Firefox does not apply templates to attributes
eyelidlessness Feb 19, 2023
4227fe4
Fix: dynamic itemset itext translations
eyelidlessness Feb 19, 2023
3d93e1b
Fix: transform appearance classes in transformer.ts
eyelidlessness Feb 19, 2023
538bcfe
Add web demo
eyelidlessness Feb 21, 2023
d9ae798
Web test/benchmark setup refinements
eyelidlessness Feb 21, 2023
97c1513
Update README with web details
eyelidlessness Feb 21, 2023
9a4a1a1
Fix build/test in CI, improve types somewhat
eyelidlessness Feb 21, 2023
5c6f898
Fix: bundle for web, preserve modules for Node
eyelidlessness Feb 21, 2023
fdb904c
Move redundant config stuff to shared module again
eyelidlessness Feb 21, 2023
77eb952
Attempt to report summaries to PR
eyelidlessness Feb 21, 2023
51595b4
GitHub does not allow forks to write PR comments at all
eyelidlessness Feb 22, 2023
69109d0
Fix build paths (hopefully for good this time!)
eyelidlessness Mar 1, 2023
50d2f76
Don't show "Transforming..." in demo when not transforming
eyelidlessness Mar 1, 2023
13b1e9c
Address initial review feedback
eyelidlessness Mar 2, 2023
885eac7
Fix: isolation of global DOM types
eyelidlessness Mar 2, 2023
e480bfe
Process itemset itext labels in XForm doc, rather than injecting temp…
eyelidlessness Mar 2, 2023
671d5c6
Use single node XPath where appropriate
eyelidlessness Mar 2, 2023
6efd980
Demo tweaks
eyelidlessness Mar 3, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 34 additions & 6 deletions .eslintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@
"ecmaVersion": 2020
},
"settings": {
"import/extensions": [".ts"],
"import/extensions": [".ts", ".tsx"],
"import/parsers": {
"@typescript-eslint/parser": [".ts"]
"@typescript-eslint/parser": [".ts", ".tsx"]
},
"import/resolver": {
"node": {
"extensions": [".js", ".ts"]
"extensions": [".js", ".ts", ".tsx"]
}
},
"jsdoc": {
Expand All @@ -48,9 +48,9 @@
{
"devDependencies": [
"app.js",
"app.ts",
"vite.config.ts",
"src/api.ts",
"src/app.ts",
"test/**/*.ts"
],
"optionalDependencies": false,
Expand Down Expand Up @@ -93,12 +93,21 @@
},

{
"files": ["./**/*.ts"],
"files": ["./**/*.ts", "./demo/**/*.tsx"],
"rules": {
"consistent-return": "off",
"no-useless-constructor": "off",
"@typescript-eslint/no-useless-constructor": "error",
"no-unused-vars": "off",
"@typescript-eslint/no-unused-vars": "error",
"@typescript-eslint/no-unused-vars": [
"error",
{
"argsIgnorePattern": "^_",
"varsIgnorePattern": "^_"
}
],
"no-dupe-class-members": "off",
"@typescript-eslint/no-dupe-class-members": "error",
"no-empty-function": "off",
"@typescript-eslint/no-empty-function": "error",
"no-undef": "off",
Expand All @@ -110,6 +119,7 @@
"error",
"ignorePackages",
{
"": "never",
"js": "never",
"jsx": "never",
"ts": "never",
Expand All @@ -119,6 +129,23 @@
}
},

{
"env": {
"browser": true,
"node": false
},
"files": ["./demo/**/*"],
"rules": {
"react/destructuring-assignment": "off",
"react/jsx-filename-extension": [
"error",
{ "extensions": [".jsx", ".tsx"] }
],
"react/no-unknown-property": "off",
"react/react-in-jsx-scope": "off"
}
},

{
"files": ["./test/**/*.ts"],
"rules": {
Expand All @@ -129,6 +156,7 @@
{
"files": ["./**/*.d.ts"],
"rules": {
"lines-between-class-members": "off",
"no-unused-vars": "off"
}
},
Expand Down
30 changes: 23 additions & 7 deletions .github/workflows/npmjs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,40 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
target: ['Node']
node: ['14', '16']
include:
- target: Web
node: 16
browser: Firefox
- target: Web
node: 16
browser: Chromium
- target: Web
node: 16
browser: WebKit
steps:
- uses: actions/checkout@v2
- uses: actions/cache@v2
- uses: actions/checkout@v3
- uses: actions/cache@v3
id: cache
with:
path: node_modules
path: |
node_modules
~/.cache/ms-playwright
key: ${{ matrix.node }}-${{ hashFiles('**/package-lock.json') }}
- uses: actions/setup-node@v2
- uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node }}
registry-url: https://registry.npmjs.org/
- run: npm install -g npm@^6
- if: steps.cache.outputs.cache-hit != 'true'
run: npm ci
- run: npm test
- run: npm run benchmarks
- if: github.event_name == 'release' && github.event.action == 'published' && matrix.node == '16'
- if: matrix.node == '16' && matrix.browser == 'webkit'
run: sudo npx playwright install-deps
- run: ENV=${{ matrix.target }} BROWSER=${{ matrix.browser }} npm test
- id: benchmarks
run: ENV=${{ matrix.target }} BROWSER=${{ matrix.browser }} npm run benchmarks
- if: github.event_name == 'release' && github.event.action == 'published' && matrix.node == '16' && matrix.target == 'node'
run: npm publish
env:
NODE_AUTH_TOKEN: ${{secrets.npm_token}}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ Thumbs.db
test-coverage
coverage.shield.badge.md
dist
.benchmarks.md
8 changes: 6 additions & 2 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,10 @@
"editor.defaultFormatter": "esbenp.prettier-vscode",
"editor.formatOnSave": true
},

"[typescriptreact]": {
"editor.defaultFormatter": "esbenp.prettier-vscode",
"editor.formatOnSave": true
},
// Note: it would be nice to reverse this! But it's currently preventing a *ton* of whitespace diff noise
"[xml]": {
"editor.formatOnSave": false
Expand All @@ -32,5 +35,6 @@
},

// Code navigation
"javascript.referencesCodeLens.enabled": true
"javascript.referencesCodeLens.enabled": true,
"typescript.tsdk": "node_modules/typescript/lib"
}
77 changes: 63 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,25 +13,22 @@ ODK XForms are based off of [W3C XForms](https://en.wikipedia.org/wiki/XForms) w

Historically, forms with many questions or many translations were prohibitively slow to transform. Starting in Enketo Transformer v2.2.1 (Feb 2023), they are much faster.

In v2.3.0 (Mar 2023), a web compatibility layer was introduced so that Enketo Transformer can be run in either a web browser using native DOM/web APIs, or in Node using a partial DOM compatibility layer wrapping equivalent `libxmljs` APIs/behavior. Each respective implementation is aliased as `enketo-transformer/dom`, resolved at build time to `src/dom/web/index.ts` or `src/dom/node/index.ts` respectively. Interfaces for the subset of DOM APIs in use are defined in `src/dom/abstract`, which ensures the Node compatibility layer conforms to the same browser-native interfaces.

Our current primary goals are:
- Using standard DOM APIs so the transformation can be performed client-side.
- Identifying and addressing remaining performance bottlenecks to remove the need for server-side caching.

Longer term, we intend to rethink transformation to be as minimal as possible, ideally without XSLT.
- Rethink transformation to be as minimal as possible, ideally without XSLT, and moving most (or all) of Enketo Transformer's current responsibilities to other parts of the Enketo stack.
- Identifying and addressing remaining performance bottlenecks to remove the need for server-side caching.

### Prerequisites

1. Volta (optional, but recommended)
1. Node.js 16 and npm 6 (Node.js 14 is also supported)

### Install as module

```bash
npm install enketo-transformer --save
```

### Use as module

#### Node

```ts
import { transform } from 'enketo-transformer';

Expand Down Expand Up @@ -60,7 +57,45 @@ const result = await transform({
// ... do something with result
```

#### Web

Enketo Transformer may also be used on the web as an ESM module. It is exported as `enketo-transformer/web`:

```ts
import { transform } from 'enketo-transformer/web';

const xformResponse = await fetch('https://url/to/xform.xml');
const xform = await xformResponse.text();
const result = await transform({
xform,
// ...
});
```

**Note:** because `preprocess` depends on `libxmljs` which is only available for Node, `preprocess` is also not supported on the web. If you must preprocess an XForm before it is transformed, you may do that before calling `transform`.

### Development/local usage

#### Install

```sh
npm install
```

#### Interactive web demo

Enketo Transformer provides a simple web demo which allows you to select any of the XForms used as fixtures in its test suites to view their transformed output, as well as toggling several of the available transform options to see how they affect the transform. To run the demo:

```sh
cd ./demo
npm install
npm run demo
```

This will print out the demo URL (typically `http://localhost:3000`, unless that port is already in use).

#### Test/dev server

Enketo Transformer provides a simple server API. It may be used for testing locally, but isn't a robust or secure server implementation so it should not be used in production. You can start it in a local dev environment by running:

```sh
Expand All @@ -84,7 +119,7 @@ sample POST request:
curl -d "xform=<xform>x</xform>&theme=plain&media[myfile.png]=/path/to/somefile.png&media[this]=that" http://localhost:8085/transform
```

#### Response format
**Response format:**

```json
{
Expand All @@ -95,7 +130,21 @@ curl -d "xform=<xform>x</xform>&theme=plain&media[myfile.png]=/path/to/somefile.
}
```

### Test
### How Enketo Transformer is used by other Enketo projects

Enketo Core uses the `transform` function directly to transform XForm fixtures used in development and test modes. It also currently uses the test/dev server in development mode to transform external XForms. It does not currently use any transformation functionality in production.

Enketo Express uses the `transform` function to serve requests to its server-side transformation API endpoints, and caches transformed XForms in Redis. It also uses the `escapeURLPath` function (implemented in `url.ts`).

Neither project currently uses the following functionality:

- Media URL mapping. Enketo Express has its own implementation of this functionality, so that dynamic media replacements are not cached. This functionality is maintained for backwards compatibility.

- The `openclinica` flag. This functionality is used by OpenClinica's fork of Enketo Express.

- The deprecated `preprocess` option. This functionality _may_ be used to update XForms with deprecated content, but its use is discouraged as users can achieve the same thing by preprocessing their XForms before calling `transform`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"with deprecated content" -> "with preprocessed content"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

custom content? I don't think "deprecated" is right but I'm not entirely sure of the intent!

Copy link
Contributor

@lognaturel lognaturel Mar 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relates to test that use preprocess and show an example of continuing to support deprecated form features. Remove the "what it's used for" part and just recommend doing any XForms preprocessing externally.


#### Test

- run tests with `npm test`
- run tests in watch mode with `npm run test:watch`
Expand All @@ -115,7 +164,7 @@ Optionally, you can add a keyboard shortcut to select launch tasks:
2. Search for `workbench.action.debug.selectandstart`
3. Click the + button to add your preferred keybinding keybinding

### Develop
#### Develop

The script `npm run develop` runs the app on port 8085 and also serves test/forms on port 8081. You could test the transformation output by placing an XForm in test/forms and running
http://localhost:8085/transform?xform=http://localhost:8081/autocomplete.xml
Expand All @@ -129,7 +178,7 @@ A vagrant configuration file and provisioning script is also included. Use DEBUG
DEBUG=api,transformer,markdown,language node app.js
```

### Release
#### Release

Releases are done each time a dependent tool needs an `enketo-transformer` change.

Expand All @@ -153,7 +202,7 @@ Releases are done each time a dependent tool needs an `enketo-transformer` chang

See [license document](./LICENSE).

In addition, any product that uses enketo-transformer or parts thereof is required to have a "Powered by Enketo" footer, according to the specifications below, on all screens in which the output of enketo-xslt, or parts thereof, are used, unless explicity exempted from this requirement by Enketo LLC in writing. Partners and sponsors of the Enketo Project, listed on [https://enketo.org/#about](https://enketo.org/#about) and on [https://github.com/enketo/enketo-core#sponsors](https://github.com/enketo/enketo-core#sponsors) are exempted from this requirements and so are contributors listed in [package.json](./package.json).
In addition, any product that uses enketo-transformer or parts thereof is required to have a "Powered by Enketo" footer, according to the specifications below, on all screens in which the output of enketo-transformer, or parts thereof, are used, unless explicity exempted from this requirement by Enketo LLC in writing. Partners and sponsors of the Enketo Project, listed on [https://enketo.org/#about](https://enketo.org/#about) and on [https://github.com/enketo/enketo-core#sponsors](https://github.com/enketo/enketo-core#sponsors) are exempted from this requirements and so are contributors listed in [package.json](./package.json).

The aim of this requirement is to force adopters to give something back to the Enketo project, by at least spreading the word and thereby encouraging further adoption.

Expand Down
64 changes: 1 addition & 63 deletions app.js
Original file line number Diff line number Diff line change
@@ -1,63 +1 @@
// @ts-check

import { createServer } from 'vite';
import { VitePluginNode } from 'vite-plugin-node';
import {
config,
external,
resolvePath,
rootDir,
} from './config/build.shared.js';

const appPath = resolvePath('./app.ts');

const init = async () => {
/** @type {import('vite').UserConfig} */
const baseOptions = {
mode: 'development',
build: {
rollupOptions: {
external,
},
},
optimizeDeps: {
disabled: true,
},
root: rootDir,
ssr: {
target: 'node',
},
};

const servers = await Promise.all([
createServer({
...baseOptions,
configFile: false,
plugins: VitePluginNode({
adapter: 'express',
appPath,
exportName: 'app',
tsCompiler: 'esbuild',
}),
server: {
port: config.port,
},
}),
createServer({
...baseOptions,
configFile: false,
publicDir: resolvePath('./test/forms'),
server: {
port: 8081,
},
}),
]);

await Promise.all(servers.map((server) => server.listen()));

servers.forEach((server) => {
server.printUrls();
});
};

init();
import './dist/enketo-transformer/app.cjs';
7 changes: 0 additions & 7 deletions config/build.shared.d.ts

This file was deleted.

Loading