Linux CI

[KOLANICH]

It'd be nice to have prebuilt wheels for Linux. So we need a linux CI.

registry.gitlab.com/kolanich/python_graal_docker:latest may be useful as a Docker image for that (includes python, GraalVM + some tools). The version from pypi works on it, to build the stuff from this repo #362 ant is needed, so you need to create a new image with all the needed build tools.

[Thrameos]

Do you want to try to set this up? I have no experience setting up CI.

[KOLANICH]

I have no experience setting up CI.

All you need is some knowledge in YAML, some knowledge in GitLab CI, some knowledge of bash, a docker image wit the most tools you need and packages for the tools not present in images.

Do you want to try to set this up?

I definitely can and 'd like to do it, but for now I have no time for it. So no ETA.

[Thrameos]

Okay sounds good. I guess the first step would be to get our existing CI to push the wheels it builds for python 2.7 and python 3.6 pushed back to github. We have a bash script for cygwin on appveyor and windows shell script. We have a second set of scripts for travis which produces some flavor of linux outputs. We just do not push anything but the test results from those builds. It I knew what items I need to push and where to push them I think I can make that happen. However, those builds are for pull requests as opposed to the current master. Is there a way to trigger a CI test of the master so we can publish the release? If we had a setup with the outline of the process (or pointer to a project example with such a thing), it would help.

I am also hoping to get the CI to start checking some other JVM. Currently we test openJDK and oracleJDK. Now that people are trying to run Amazon it would be good to start testing those as well.

[KOLANICH]

I guess the first step would be to get our existing CI to push the wheels it builds for python 2.7 and python 3.6 pushed back to github

IMHO: we don't need push wheels on GH - it would require giving a CI the credentials to access the repo. And these credentials will likely be compromised because "s" in the word "Docker" stands for "security".

Instead the nightly packages should be used directly from CI. Some CIs allow direct links to the latest artifacts. These links should be used. Uploading to PyPi from CI is also not needed. Instead JPype1 @ https://.../...whl syntax should be recommended. And for distros the stuff should be installed using distros package manager, not pip. So we also need to create packages for distros from wheels. I have created some tools to create deb-packages and distros from prebuilt distributions of software using python, because bash scripting and debhelper really suck. They may be useful. Also I have some packages built with these scripts, such as GraalVM and CMake.

And the most important point - all the official releases should be build manually by a human on own machine, checked and digitally signed. The builds via CI should be also signed, but using the different key in which name it should be stated explicitly not to trust it much.

[Thrameos]

Not sure if this helps. https://www.appveyor.com/docs/deployment/github/ https://docs.travis-ci.com/user/deployment/releases/ I do see that it requires a secure key. Not sure how to create it or use it. It looks like adding something to travis and appveyor file would allow us to push an artifact to github that we can then associate with the release. To deploy we would add a tag to the master and publish it to start the process. The artifacts get built, then we go back to github and make the release official which would put those binaries out there. We can then push the binaries to another hosting service for download. Is that a correct view?

[KOLANICH]

I do see that it requires a secure key. Not sure how to create it or use it. It looks like adding something to travis and appveyor file would allow us to push an artifact to github that we can then associate with the release.

It will, but it will compromise the credentials and with credentials the account. I strongly advise against it. If an adversary compromised the credentials and uploaded a backdoored version, IMHO it would be a disaster.

To deploy we would add a tag to the master and publish it to start the process.

I advise against it. IMHO releases should be built manually and digitally signed by the developer. The point here - a developer is in better control of an own machine than a free CI maintained by a third party and executing the code from anyone. If this code can break isolation (there were multiple examples of breaking isolation in past, from breaking containers like Docker to breaking hypervisors using microarchitecturial attacks like flip feng-shui or just extracting the info using side channels, like cache attacks).

[Thrameos]

It will be difficult to make binaries for me. My primary machine can’t be used for creation of binaries due to policy even under incidental use. I have no control over the build environment on the supercomputers, and even if I could those machines are not very typical. I agree that using travis and appveyor would run serious security risks. Thus we need a some other way to generate binary distributions.

[KOLANICH]

It will be difficult to make binaries for me. My primary machine can’t be used for creation of binaries due to policy even under incidental use.

How about other devs with write access to this repo (I guess if they have direct write access they are trusted enough to make releases)?

[marscher]

I would vote to ship binaries as manylinux wheels on PyPI. This way jpype will be easily installable via pip without the compilation hassle. There is a manylinux docker image, to which we can add ant and a jdk during runtime (no need to define our own container).

@Thrameos I see your concerns about compromising CI, but think of the likelihood of doing so. One would need to know on which host machine a build will be scheduled and then try to break in within minutes (before the actual build is done and uploaded). The whole conda-forge channel is being built on public CI. If you opt out automatizing this process who should be in charge of compiling the package and upload it?

[marscher]

pypa/manylinux#118 BTW shipping a c++ based package is much more complicated than a plain c based one.

[Thrameos]

@marscher KONLANICH is strongly opposed to building binaries using the CI. My main concern is who does the build and and hard is it to produce. Automatically produced binaries using a trusted method supported by GitHub would be best assuming the image can't be compromised. If building binaries requires a lot of effort it won't get done. But given CI isn't my area I don't know what GitHub has as far as building binaries, and at the same time we are constantly getting the CI images which we use changing out from under current system. If the build system had a fixed image for each architecture we want it would certainly help. We could place checks like building md5sums of the source that was being built, but if the image were compromised that would be just as untrustworthy. So I don't really no what to do here other than recruit someone to the project to handle binary builds if we can't build it automatically.

There are certainly security concerns that can be raised and we shouldn't dismiss out of hand. My site for example prohibits any installation of software binaries without approval and strongly prefers build from source when dealing with open source products. But then I work on super computers processing nuclear data, so my site is certainly on the paranoid end of the scale.

[KOLANICH]

This way jpype will be easily installable via pip without the compilation hassle.

I also vote for building and shipping binaries, but not via pypi. PyPi design is IMHO flawed - it has no guarantees that the binaries put there correspond to sources in VCS, it has no guarantees of packages origin and integrity and it is prone for typesquatting been used to spread malware. So currently CIs are more trustworthy in some sense because they build packages from the code publicly available in the repo. Of course CIs can be compromised themselves, so we need reproducible builds, allowing to verify that neither CI nor Docker images used add something unauthorized to binaries.

I see your concerns about compromising CI, but think of the likelihood of doing so. One would need to know on which host machine a build will be scheduled and then try to break in within minutes (before the actual build is done and uploaded).

One only needs to get root access to CI (break virtualization solution used) and then inspect all the containers for secrets or modify all the containers the way making them to add malware or a backdoor into all the binaries built. Of course it is not easy, but definitely not impossible.

@KOLANICH is strongly opposed to building binaries using the CI.

I'm not. I am strongly opposed of putting sensitive information to CI, like credentials to access the repo. I am also against building RELEASE builds on CI, because from release builds greater quaility is expected rather than from nightlies, and this quality includes security. Releases are rare, it is definitely affordable to build and check an sign them manually. Especially taking in account that every active dev builds the binaries locally a lot in order to test and debug them. Nightly builds of course should be fully automated.

[Thrameos]

Sorry if I mischaracterized your position. However, it still leaves us with the position that binaries would be nice to have, but we still lack a good plan on how to get them. When I worked for a university it wasn’t an issue to set up a script to launch jobs on a bunch of machines to produce binaries. As it stands currently, we are relying on 3rd party packagers such as the anaconda maintainers to create the binaries.

[marscher]

I'am the conda-forge "maintainer", e.g. one person with commit rights to the feedstock ;)

[Thrameos]

Closing. Now built using docker.