Making a single-file mermaid-cli.exe for Windows

[ekaterina-o]

Is your feature request related to a problem? Please describe.
I have a few diagrams in md file and would like to convert all of them at once to svg.

Describe the solution you'd like
I would like to have something like pandoc app I can install with exe on Windows 7 and then use with command line to convert md file with mermaid code to new md file with linked svg images of diagrams. I don't want to install npm etc, for existing mermaid CLI because I'm not a programmer.

Describe alternatives you've considered
Now I convert every diagram manually in online Live Editor. It's very inconvenient.

[rvkennedy]

It is an awkward setup for anyone who doesn't use node.js day-to-day. You need to :

1. Install node and npm https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
2. Install mermaid-cli using 'npm install -g @mermaid-js/mermaid-cli'
3. Locate where npm has installed mmdc.cmd - for me it's in C:\Users\[MY USERNAME]\AppData\Roaming\npm\mmdc.cmd
4. Set mermaid_cmd in conf.py to point to that cmd file, e.g. mermaid_cmd='C:\Users\[MY USERNAME]\AppData\Roaming\npm\mmdc.cmd'
5. Install the Sphinx Mermaid support with 'pip install sphinxcontrib-mermaid'
6. Add "sphinxcontrib.mermaid" to extensions in conf.py
7. Set mermaid_output_format to 'png' - 'raw' doesn't work for me at least.

And finally it should generate diagrams.

[aloisklink]

This should be relatively straightforward to setup with https://www.npmjs.com/package/pkg Edit: seems like pkg doesn't yet support ESM, see vercel/pkg#1291.

We could set up a GitHub Action that runs automatically and uploads the .exes as a GitHub release asset.

The only hard part bit that might be complicated is puppeteer, which automatically downloads a ~350 MiB chromium instance at run time.

Unfortunately, I don't have a Windows machine so it would be difficult for me to test the actual .exe file that is generated, nor do I have much Windows experience. But I wouldn't mind reviewing the code if a community member made a PR. I'd imagine it needs to look something like:

• GitHub Action that runs on every commit that:
  • Builds .exes with https://www.npmjs.com/package/pkg (or some other tool)
  • Tests the .exe in Windows (maybe also uploading some of the generated diagrams)
  • Uploads the .exe as a CI artifact for people to download
• GitHub Action that runs on every release that:
  • Uploads the .exe as a release asset for people to download

@MindaugasLaganeckas, if somebody made a PR that implemented the above, would you be happy for it to be added? (if so, feel free to add the help-wanted label).

[MindaugasLaganeckas]

Thank you for a good discussion. I have limited access to Windows as well, but fortunately Github provides windows based build agents. Yes, I would accepts such PR given that some e2e/integration tasks come with the PR.

[Stephanevg]

This should be relatively straightforward to setup with https://www.npmjs.com/package/pkg.

We could set up a GitHub Action that runs automatically and uploads the .exes as a GitHub release asset.

The only hard part bit that might be complicated is puppeteer, which automatically downloads a ~350 MiB chromium instance at run time.

Unfortunately, I don't have a Windows machine so it would be difficult for me to test the actual .exe file that is generated, nor do I have much Windows experience. But I wouldn't mind reviewing the code if a community member made a PR. I'd imagine it needs to look something like:

• GitHub Action that runs on every commit that:

  • Builds .exes with https://www.npmjs.com/package/pkg (or some other tool)
  • Tests the .exe in Windows (maybe also uploading some of the generated diagrams)
  • Uploads the .exe as a CI artifact for people to download

• GitHub Action that runs on every release that:

  • Uploads the .exe as a release asset for people to download

@MindaugasLaganeckas, if somebody made a PR that implemented the above, would you be happy for it to be added? (if so, feel free to add the help-wanted label).

Hi guys, I would have an interest in having a mermaid-cli agent available on Windows systems, and would be potentially interested in taking this one over, but my experience with building npm is unexistent. (the github action stuff i guess I can learn it on the fly).

In order to develop this, I guess I'll create a fork of the project, and would be able to test things like uploading the CI artifact and the release asset.

For me to actually be able to do this, I would really need to have the detailed steps of what I need to do, and in what order.

Also, I would recommend we only build the arrtifacts on successfull pull requests, instead of every commit. That would become quite messy otherwise ;)

Tasks

Regarding this

• GitHub Action that runs on every commit that:

  • Builds .exes with https://www.npmjs.com/package/pkg (or some other tool)
  • Tests the .exe in Windows (maybe also uploading some of the generated diagrams)
  • Uploads the .exe as a CI artifact for people to download

build

I currently don't know how to do this. Could someone maybe provide that to me? (@MindaugasLaganeckas / @aloisklink maybe ?) I can than integrate it in a github action.

tests

There are two ways to go forward with this:
a) either I can write a new set of tests using pester (the powershell / windows unit testing framework)
b) or try to reuse the existing set of tests that already exists.

I would prefer option b, as it make sense to me to reuse as much as possible.
Or maybe we could organize everything in a way, that I use sources that are already tested, and that I write only 'installation' / 'execution' tests for Windows (I create for example a basic diagram, and validate it doesn't fails, and it contains the html elements that it is suppose to have. (That would be option C i guess...).

Artifact upload

I only have a few doubts on how to test this as I have no rights on this current project. I assume that I would be able to try this out want once I fork this project, right?

I do have some difficulties to see the difference between the artifact upload, and the release upload. They look identical to me tbh..

Let me know if you guys are for this :)

[MindaugasLaganeckas]

Thank you for reaching out ❤️ I would suggest to start with local testing (on your computer). And reuse as much as possible. I think option C is the one to go for:

• reuse sources/binaries that are already tested
• fabricate a windows executable from the above
• implement worlds simplest test with one simple diagram to verify that the new executable actually works (no need to re-test mermaid-cli functionality)
• I can help you with the pipelines. But first prepare the steps above locally on your computer 😄
• when you are ready, we will implement a new pipeline that is executed by a windows agent and connected to the overall flow
• I suggest that the executable will be built as part of the release flow (no feature builds) and will be available for downloads from the github release artifacts

What do you think?

[aloisklink]

build

I've never done anything like this before, but I think there's two approaches that might work:

• The pkg library: https://www.npmjs.com/package/pkg
• Or Node v20 has it's own feature for making this: https://nodejs.org/api/single-executable-applications.html

The only problem I see with both of them is that they currently only work with CommonJS JavaScript libraries. Since mermaid-cli and it's dependencies use ESM, you'd have to somehow transpile ESM to CommonJS, i.e. with esbuild.

Otherwise, there might be another tool somewhere that can work with ESM code directly!

In that case, I'd imagine it would something like:

• npm install (install mermaid-cli dependencies)
• npm install pkg (install packages need for Windows EXE creation)
• vite | rollup | esbuild (what ever command you need to create a single mmdc-single-file.cjs bundle)
• npx pkg mmdc-single-file.cjs --target node18-windows-x86 --output "mmdc-${VERSION}-x86_64.exe"

I know this stuff is pretty JavaScript specific, so if you need help with this bit, let me know! I think I can probably get an executable working on Linux at least, and then hopefully you'll be able to modify the commands to get it working on Windows!

tests

Agree with @MindaugasLaganeckas :) I think as long as it works with a single diagram, we can count it as a success! You can use actions/upload-artifact in GitHub Actions to upload the generated image, so something like:

  - run: |
      my-mermaid-file.exe -i input.mmd -o output.png
  - uses: actions/upload-artifact@v3
    with:
      name: windows-exe-output.png
      path: output.png

Artifact upload

Build artifacts are uploaded using GitHub Actions with the actions/upload-artifact action, and you can test it in your own GitHub fork :) They upload to the Github Actions run UI, e.g. https://github.com/mermaid-js/mermaid-cli/actions/runs/4785247117. These are useful for debugging, or for power users that want to test the latest version of a program (old artifacts are even auto-deleted).

Release assets are uploaded using the GitHub API (maybe using a GitHub Action?). You can view these at https://github.com/mermaid-js/mermaid-cli/releases/tag/10.1.0. These are the versions we expect most people to download.

I'd imagine you'd have two GitHub Actions:

• One runs on every push or pull_request (can be modified in https://github.com/mermaid-js/mermaid-cli/blob/master/.github/workflows/test.yml)
  This will build the .exe, do a basic test of the .exe, and upload the .exe as a build artifact using actions/upload-artifact.
• The other action runs on every release (can modify https://github.com/mermaid-js/mermaid-cli/blob/master/.github/workflows/release-publish.yml).
  This will build the .exe (can skip tests), then upload it as a release asset, maybe using GitHub CLI's tool, something like:
  gh release upload "${{env.RELEASE_VERSION}}" "mmdc-${{env.RELEASE_VERSION}}-x84_64.exe" (see https://cli.github.com/manual/gh_release_upload)
  • This one is a bit more annoying to test, but since it should mostly be copied over from the first action, so it's not too bad.

[Stephanevg]

Hey guys I finaly found some time to look at this.
I do believe I have managed to identify the working process, but it looks like that I am still missing something, as it ultimatley does not generate the .mmd.svg file.

In a nutshell, this is what I am doing (based on the above suggestions).

1. Install nodejs
2. install -g npm
3. install -g pkg
4. npm install -g @mermaid-js/mermaid-cli
5. cd C:\Users\MyUser\AppData\Roaming\npm\node_modules@mermaid-js\mermaid-cli
6. pkg . --out-path C:\Users\Stephane\OneDrive\PARA\Projects\MermaidJs-cli\

When executing the pkg command, I do have the following errors that pops up:

PS C:\Users\Stephane\AppData\Roaming\npm\node_modules\@mermaid-js\mermaid-cli> pkg . --out-path C:\Users\Stephane\OneDrive\PARA\Projects\MermaidJs-cli\
> pkg@5.8.1
> Targets not specified. Assuming:
  node18-linux-x64, node18-macos-x64, node18-win-x64
> Warning Babel parse has failed: import.meta may appear only with 'sourceType: "module"' (10:30)
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules\puppeteer\.local-chromium
  %2: path-to-executable/puppeteer
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules\puppeteer\.local-chromium
  %2: path-to-executable/puppeteer
> Warning Failed to make bytecode node18-x64 for file /snapshot/node_modules/@mermaid-js/mermaid-cli/src/cli.js
> Warning Failed to make bytecode node18-x64 for file /snapshot/node_modules/@mermaid-js/mermaid-cli/src/index.js
> Warning Failed to make bytecode node18-x64 for file /snapshot/node_modules/@mermaid-js/mermaid-cli/node_modules/chalk/source/index.js
> Warning Failed to make bytecode node18-x64 for file /snapshot/node_modules/@mermaid-js/mermaid-cli/node_modules/chalk/source/utilities.js
> Warning Failed to make bytecode node18-x64 for file /snapshot/node_modules/@mermaid-js/mermaid-cli/src/cli.js
> Warning Failed to make bytecode node18-x64 for file /snapshot/node_modules/@mermaid-js/mermaid-cli/src/index.js
> Warning Failed to make bytecode node18-x64 for file /snapshot/node_modules/@mermaid-js/mermaid-cli/node_modules/chalk/source/index.js
> Warning Failed to make bytecode node18-x64 for file /snapshot/node_modules/@mermaid-js/mermaid-cli/node_modules/chalk/source/utilities.js
> Warning Failed to make bytecode node18-x64 for file C:\snapshot\node_modules\@mermaid-js\mermaid-cli\src\cli.js
> Warning Failed to make bytecode node18-x64 for file C:\snapshot\node_modules\@mermaid-js\mermaid-cli\src\index.js
> Warning Failed to make bytecode node18-x64 for file C:\snapshot\node_modules\@mermaid-js\mermaid-cli\node_modules\chalk\source\index.js
> Warning Failed to make bytecode node18-x64 for file C:\snapshot\node_modules\@mermaid-js\mermaid-cli\node_modules\chalk\source\utilities.js
PS C:\Users\Stephane\AppData\Roaming\npm\node_modules\@mermaid-js\mermaid-cli> 

It still creates a set of 3 files

PS C:\Users\Stephane\OneDrive\PARA\Projects\MermaidJs-cli> dir


    Directory: C:\Users\Stephane\OneDrive\PARA\Projects\MermaidJs-cli


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---l        22/07/2023     08:45       67352823 mermaid-cli-linux
-a---l        22/07/2023     08:45       72339815 mermaid-cli-macos
-a---l        22/07/2023     08:45       58745387 mermaid-cli-win.exe
-a---l        22/07/2023     08:31            533 test.mmd

but, if I launch the creation of mermaid file, I get the following result

PS C:\Users\Stephane\OneDrive\PARA\Projects\MermaidJs-cli> .\mermaid-cli-win.exe .\test.mmd
(node:28116) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `mermaid-cli-win --trace-warnings ...` to show where the warning was created)
C:\snapshot\node_modules\@mermaid-js\mermaid-cli\src\cli.js:3
import { cli, error } from './index.js'
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:355:18)
    at wrapSafe (node:internal/modules/cjs/loader:1040:15)
    at Module._compile (node:internal/modules/cjs/loader:1076:27)
    at Module._compile (pkg/prelude/bootstrap.js:1933:32)
    at Module._extensions..js (node:internal/modules/cjs/loader:1166:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:834:12)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47

Node.js v18.5.0

using .\mermaid-cli-win.exe --trace-warnings .\test.mmd as suggested in the error message above actually outputs the same exact same error message

SO....

I am not Javascript savy, so when I read this error message, I feel like an elderly person using a computer for the first time, and trying to understand the difference between the right and the left click. :D

aloisklink, would your offer for help still stand? I think I could need some ;)

[aloisklink]

SyntaxError: Cannot use import statement outside a module

It sounds like you're running into the following issue:

The only problem I see with both of them is that they currently only work with CommonJS JavaScript libraries. Since mermaid-cli and it's dependencies use ESM, you'd have to somehow transpile ESM to CommonJS, i.e. with esbuild.

Basically, mermaid-cli uses a system of packaging files called ESM (for ECMAScript modules), which is the official module specification for JavaScript. It can be recognized by import abc from "abc"; statements. However, since ESM was only recently added to the JavaScript standard, Node.JS invented their own module system years ago called CJS/CommonJS (can be recognized by const abc = require("abc");).

Everybody is slowly moving away from the old CommonJS standard (we had to move away because our dependencies moved to ESM). However, a bunch of older tools still don't support it.

I've tried using esbuild to convert from ESM to CJS (see aloisklink@ed0880c for my WIP), but it looks like pkg still can't understand CJS converted from ESM.

My feeling is that if we want to use pkg, we need to wait until vercel/pkg#1291 is fixed.

It might be possible to do this currently with electron, but I think it will be a lot of work, so it's probably better to wait for pkg to support ESM, unless you're willing to spend a lot of time experimenting.

[Stephanevg]

Thanks @aloisklink for the quick explanation 👍

My feeling is that if we want to use pkg, we need to wait until vercel/pkg#1291 is fixed.

I believe that that would be the wise solution, indeed. But on another side, I am a bit afraid that this might be a long wait.
I would love to help, but my javascript skills are pretty low. I am willing to learn, but I feel like this project might be the wrong to sharpen my javascript skills, as I feel one needs to have a deep understanding of the core Javascript concepts.

I'ill wait for that pkg issue to be fixed, or hope that another person can come up with a solution on how to do that.

Once that is done, I believe I have figured out how to automate the process on every new release and attach it as a release.

[aloisklink]

I believe that that would be the wise solution, indeed. But on another side, I am a bit afraid that this might be a long wait.

There is an experimental Node.JS single-executable application API (see https://nodejs.org/api/single-executable-applications.html#single-executable-applications) that seems like it will help, but it also currently only supports CommonJS and hasn't yet added ESM support (plus, the API is designed for libraries like pkg, not users like us). So even if pkg doesn't add ESM support soon, hopefully at least Node.JS SEA will!

[aloisklink]

Apparently both Bun and Deno support creating a single-file executable with ESM support

I see that both Deno and Bun can create executables with ESM support.

• https://deno.land/manual@v1.36.0/tools/compiler
• https://bun.sh/docs/bundler/executables

See vercel/pkg#1291 (comment)

I've got no idea if meramid-cli and it's dependencies will work with Bun/Deno, but if it does, that's one potential way to create a single file .exe!

[Stephanevg]

Thanks for the hints @aloisklink

I have looked at both options. Bun currently only offers limited support on Windows. Since we are trying to build a binary for Windows I will skip this one.

I have looked a bit at deno.

I have tried to compile it as this

deno compile .\@mermaid-js\

but i get the following error:

error: Access is denied. (os error 5)

I quickly googled it, and it mentions some missing .ts references. Not sure which direction to go from here...
Do you have any suggestions @aloisklink ?

[aloisklink]

I quickly googled it, and it mentions some missing .ts references. Not sure which direction to go from here...
Do you have any suggestions @aloisklink ?

Sorry, I've never used Deno or Bun before, so I don't know. I don't even know if the mermaid-cli package would run on Deno (because we use puppeteer, mermaid-cli is a bit more advanced than the average Node.JS package).

You might find it a bit easier to try building the .exe on Linux and cross-compile it for Windows. Generally speaking, the developer experience for open-source is easier on UNIX systems like Linux, so it's possible that Deno is more stable on Linux too.

A good first step might be to make confirm that mermaid-cli actually works on deno on Windows, and only then try to start making a single-file .exe.

[Stephanevg]

You might find it a bit easier to try building the .exe on Linux and cross-compile it for Windows. Generally speaking, the developer experience for open-source is easier on UNIX systems like Linux, so it's possible that Deno is more stable on Linux too.

I would actually be ok in generating the .exe on linux, and than 'cross-compile' it. Ultimatley, the end goal is to have a working CLI solution working on Windows. We don't really care where / how we create that exe. As long that it works... ^^,

I am not familar with that. Would you know I would go to do that?.

[Stephanevg]

I also asked ChatGPT, it mentionned tools I we have not address here yet (but maybe you guys ruled them out for obvious reasons which I don't know about ?)

• nexe
• node-packer (nodec)

It also mentionned pkg (Which we already disqualified) and electron.

nexe

I quickly had a lookd a nexe and it looks like you have to convert a .js file. The examples show an http application which is contained in a .js file, and simply 'piped' to nexe. I don't know how dependencies (and ESM) would be handled in there. Any one has experience with this tool already? (But it looks like node-packer can actually do more - so continue reading ;) )

node-packer (nodec)

The node-packer github project sounds promising.

I specifically like this table where it compares the solution to other tools.

It works on Windows directly apparently, although this might not necessarly be needed since It also seems it could be a solution for cross compiling as mentionned here.
It looks like using the --dest-arch=x64 could actually generate a windows x64 executable.

do you @aloisklink or anyone else has experience with this? What you guys think of this potential approche?

[aloisklink]

I would actually be ok in generating the .exe on linux, and than 'cross-compile' it. Ultimatley, the end goal is to have a working CLI solution working on Windows. We don't really care where / how we create that exe. As long that it works... ^^,

I am not familar with that. Would you know I would go to do that?.

It depends on which program you are using. pkg by default creates a Linux, macOS, and Windows .exe, but you can limit it to just creating a Windows version by specifying --targets node20-win-x64. See the Targets section in their README.md: https://github.com/vercel/pkg#targets ¹. deno compile also accepts a similar --target CLI flag, see https://docs.deno.com/runtime/manual/tools/compiler#cross-compilation.

---

• nexe

nexe doesn't support ESM yet: nexe/nexe#815

However, you might have luck using aloisklink@ed0880c, which is my WIP commit that creates a single-file .js file at packages/mermaid-cli-single-file/dist/app.js. pkg unfortunately couldn't understand this file, but maybe nexe might?

• node-packer (nodec)

The fact that node-packer hasn't been updated since 2021-01-03 means I think it's pretty unlikely to work with ESM. Because it uses SquashFS, it might work with my WIP single-file .js in aloisklink@ed0880c, but 🤷. I'll be honest, even if it does work, do we really want to use a package that hasn't been updated in 3 years? What happens if there are security issues/bugs?

Potential alternatives:

• caxa might work, since apparently it does support ESM: FYI about ESM leafac/caxa#2. Apparently, because it uses a "self-extracting archive", it might be even easier to get it working properly with puppeteer: https://github.com/leafac/caxa#the-solution.
• electron will support ESM in their next Electron v28 version: Support Node's ES Modules electron/electron#21457. FYI, I think this will be a lot more work to get working, since Electron is designed for proper desktop applications (e.g. like Slack/Discord/Microsoft Teams), but it should definitely work (although it might be waaay too much work to get it working in the first place!). My advice is to only go for this if you want to create a nice user friendly interface, and don't mind spending a looong time working on it.

Footnotes

1. Apparently, a Linux server can also cross-compile an arm64 Windows .exe executable, even though an x86 Windows server can't create an arm64 Windows .exe, so it's actually better to use Linux to build the .exe! ↩

[ThomasTNO]

Any progress on this?

[Stephanevg]

Hey @aloisklink I am reactivting this threat. As mentionned, I am not a Javascript guru, so I don't know what happened in the last year or so in that field, but I can imagine that 'some stuff' has changed, right?

Do you know if the ecosystem has evolved in a way where we would have a technical solution to potential implement this one ? (I would still be happy to implement it - just need some guidance ... ;) )

[aloisklink]

but I can imagine that 'some stuff' has changed, right?

Unfortunately, I think in the Node.JS eco-system, it might have become worse 😢

• https://github.com/nodejs/single-executable hasn't been updated since 2022.
• https://github.com/leafac/caxa has been deprecated.
• https://github.com/vercel/pkg was deprecated and archived last year.

But, I think Deno has really improved support with Node.JS packages, so maybe that's an option?
And I think Bun is meant to better too.

It could be worth a try to see if Deno can run mermaid-cli, and if so, then using deno compile doesn't look too hard.