-
Notifications
You must be signed in to change notification settings - Fork 30
[RFC] Statically linked binaries #31
Comments
Hey there! I'm really looking forward to this discussion. For creating truly static binaries, as you pointed out, musl-c needs to be used instead of glibc. This would typically involve building with the appropriate flags inside an alpine container, or at least with a GHC compiled with musl-c. I'm not sure how much heavy lifting As it is, there's no way that I can see of to really make a Separate from that, there's other things that could certainly be pseudo-standardized on, such as name triples, that would make integrating various projects easier (eg it would simplify implementation of providing a |
Thanks for your feedback @jared-w! In terms of what |
I've been continuing to think about how this could work, and I really don't see how it can.
In particular, this line is really asking to re-implement an entire CI pipeline worth of logic. After all, if you support a magical "build step", then almost immediately you'll run into a situation where this won't work. Most of the haskell projects I've seen would fail this, actually; either the build step is non-standard or the environment is, or... "Standard" just really doesn't mean much when it comes to software development environments. Taking a step back, one might consider just passing appropriate flags to GHC. Really, the "only problem" with just using the appropriate flags like But say someone really wants to use muslc and alpine approach. The ghc-musl docker images are interesting, but they can't sanely be used inside the action as an invisible abstraction. I suppose that's really the biggest dilemma. In order to really be successful, actions have to provide transparent abstractions. That just can't really happen with static binaries; they demand far too much knowledge of their environment to be abstracted away. It would be different if it was the default, like in rust or golang (this conversation wouldn't even be happening if that was the case). Building static haskell binaries as part of a CI workflow could certainly be more ergonomic, but other than codifying passing in the right flags, I don't know what else could be done to simplify things from an action's programmatic point of view. Any approach involving docker is right out and must be documented in a README that hopefully doesn't bitrot. The approach with compiler flags is a flimsy abstraction that must be kept in the back of one's mind. And maybe the simple happy path can be abstracted entirely away, but I'm unsure how much of a benefit that would even be. So, to summarize all of that up, the only sane thing I can think of That said, I think a |
Thanks for the mention @chshersh . Here's my 2c: I mostly agree @jared-w, where the hard part about static-compiling Haskell comes from setting up the appropriate environment, also that However; not knowing really how this action works, take this with a grain of salt; but I do see a workable way to the key problem @jared-w pointed out:
This is pretty much the selling point of Nix; it is pretty good at isolating/defining software in a way that it works on any environment. So, I believe below should be doable:
So, it wouldn't be trivial, but in the end the interface might be as simple as
I think this is a good idea. If anyone picks it up, I would also be happy to help if I can. |
@jared-w @utdemir Thanks a lot for your feedback! I'm excited to see this issue moving forward by discussing possible ways of implementing the feature 😊 The solution with - if: matrix.os == 'ubuntu-latest'
name: Build static binary
run: |
mkdir dist
sudo apt-get install -y musl
cabal install --install-method=copy --overwrite-policy=always --installdir=dist ${{ steps.setup-haskell.outputs.static-flags }}
- if: matrix.os != 'ubuntu-latest'
name: Build non-static binary
run: |
mkdir dist
cabal install exe:stan --install-method=copy --overwrite-policy=always --installdir=dist Btw, where can I read about the flags I need to pass to GHC to build static binaries linked with In terms of reusing I imagined the following workflow based on Docker containers:
Initially I was thinking about implementing a separate GitHub action that does exactly this. But I don't have much experience with neither Docker no TypeScript to build an action. Apparently, it's not trivial to copy files between Docker-based GitHub actions and host. I've asked similar questions in the GitHub Community:
But maybe this is a solvable problem for someone with more experience in building GitHub Actions or using TypeScript 🙂 |
This is what ghcup does:
As such, the ghc flags are just I believe alpine to be the easiest solution to this. You build a binary and ship it. How you built the binary doesn't have to be reproducible for 99% of the people. Also note that ghcup supports most GHC versions on alpine (even 32bit), so you can use ghcup to install the target versions: ghcup alpine versions
|
This will run into issues unless you're very careful about exactly what directories you pass into docker. Sharing directories where build artifacts will be created inside docker (and possibly outside of docker), particularly More importantly, It's much easier to just use a docker container from start to finish; then you avoid these problems because you're not blending different platforms together. You still have the musl vs glib issue where native code and the FFI get more difficult to work with, but at least you're not dealing with mixing abstractions in incompatible ways. Further, Github actions has support for just using a container, so theoretically I think it's possible to have an example like. jobs:
static:
runs-on: ubuntu-latest
container: node:12 # to make sure that you can download and run actions inside the container. I think?
steps:
- uses: actions/checkout@v2
- uses: actions/setup-haskell@v1 # <- warning, downloads GHC, cabal, etc., from scratch *every time*.
- .... that would, more or less, "do the right thing". (as an aside; this means setup-haskell can be used in any container, in theory. Currently it assumes it'll only be run natively in github actions and uses those assumptions to simplify things. I think it might actually still work in a container, thanks to Unfortunately, using a container for linux means you can no longer have a convenient 3-OS build matrix. The build matrix is really nice since github doesn't offer a lot of code de-duplication opportunities through traditional yaml shenanigans. Nix could potentially solve this, I think, but it would be the opposite of a transparent abstraction; nix likes to be the entire solution, not just part of it. It would also destroy CI pipeline speeds; ghc's closure size is enormous in nix and that doesn't even take into account the time required to install and setup nix from scratch every time. |
Here's an example of static binary release for linux: https://github.com/hasufell/stack2cabal/blob/master/.github/workflows/release.yaml#L32 |
@hasufell That's an amazing example! 😍 Does anyone know, if it's possible to define matrix with the container only for a single item? So some boilerplate can be removed. Something like: runs-on: ${{ matrix.os }}
container: ${{ some variable for 'alpine:3.12' only for 'ubuntu-latest', otherwise no container }}
strategy:
matrix:
os: [ubuntu-latest, macOS-latest, windows-latest]
... @hasufell Another questions. Is there a difference between |
@chshersh This is not possible, unfortunately. It's an explicit limitation of github actions that is a little annoying. More broadly, there are no top level object keys that can be optional that I'm aware of, and having values of undefined/null/falsy is almost universally an error. So even just Rust seems to avoid this by allowing you to directly use muslc regardless of the host OS. It would be interesting to see if that would be a viable path for GHC/Haskell to take, but I feel a github action is the wrong level to support something of that complexity/nature. Ideally it'd be possible in a more generic fashion.
As far as I know, they're identical. |
I'm opening this issue to start a discussion around the ability to build statically linked binaries for Haskell projects. I think it will be extremely beneficial for the whole Haskell community if developers could produce such binaries easily with GitHub Actions workflows.
The following blog post describes in detail how to produce binaries for Haskell applications on all three operating systems using the
setup-haskell
action:Cabal has the
--enable-executable-static
flag (mentioned in the blog post about the latest changes in HLS) which allows building statically linked binaries. Still, since they are built on Ubuntu, they are not truly statically linked. I expressed my concerns in the comments under the blog post:For real static binaries, you need to build them inside the Alpine-based Docker container. I've used the ghc-musl in the past, and I find it quite pleasant and easy to use, but I had to do everything manually on my laptop. It would be nice to automate this process somehow.
I'm going to mention a few people, who might be interested in this discussion (sorry for notifications, feel free to unsubscribe from this conversation):
ghc-musl
project)ghcup-hs
)I would like to hear your thoughts on how we can proceed with this!
The text was updated successfully, but these errors were encountered: