Reproducible Builds

[TeunVR]

Copy of minvws/nl-covid19-notification-app-design#10

Het vrijgeven van de source-code van de covid-notification app is belangrijk voor de acceptatie ervan.
Het is echter niet eenvoudig om te verifiëren dat de binary die in de App Store en Play Store staat ook werkelijk op basis van die source-code is gebouwd.

Andere projecten (bv Signal en Telegram) staan voor dezelfde uitdaging:
signalapp/Signal-iOS#641
https://core.telegram.org/reproducible-builds#reproducible-builds-for-ios

Helaas lijkt er niet direct een technische oplossing hiervoor te zijn.
Misschien is het een idee om dit procesmatig in te regelen.
Bv door onafhankelijke “getuigen” tijdens bouwen/submitten naar stores?

[github-actions]

Thanks for opening your first issue in this repository! We know that the process of creating an issue and pull request can sometimes be the biggest barrier for new contributors. So first of all thank you!

If this is your first time contributing to open source we recommend checking out the First Timers Only website.

🤔 What you will need to know

This automated comment is meant to welcome you to our community and we are happy to walk you through the process. In case you haven't yet, we ask you to read our Code of Conduct, and contributing documentation. If you're not super familiar with GitHub yet please feel free to follow one of the Learning Lab courses.

If you're reporting a 🐞, please make sure you include as much relevant information as possible, like a screenshot and steps to reproduce it.

⌛ What to expect

We receive a lot of input, so please be patient and we will get back to you as soon as we can! 😊

👍 Involvement

If you plan to do something more involved, please reach out to us via Slack at praatmee.codefor.nl. This will avoid unnecessary work and surely give you and us a good deal of inspiration.

[bwbroersma]

It's not only having reproducible builds, but also submitting the builds to a non-changeable log, e.g. use Certificate Transparency to publish the sha2 hash of the published build as a subdomain (with a SSL certificate). This is the ProtonMail way of doing it, see a FOSSDEM 2020 talk Building a Web App that Doesn’t Trust the Server - Securing ProtonMail, slide 8-11 and video recording from 5:07 to 11:27.

[lwesterhof]

Signal for Android has reproducible builds: https://github.com/signalapp/Signal-Android/blob/master/ReproducibleBuilds.md (https://signal.org/blog/reproducible-android/)

[TeunVR]

Yes Android looks a bit less complicated in this case.
For iOS it is pretty hard to compare the binary from the AppStore to a self-built binary.

[ruuda]

The F-Droid Android app store does support reproducible builds.

[dirkx]

Ah - lovely - so that would be a very worthwhile PR once we have the android code base up.

[dirkx]

Now that the first code repo's have gone up; with the rest to follow in the next days - I am going to close this one; and PR's for reproducible builds are more than welcome at the code repos. Hope you do not mind. And thanks for sharing and contributing this good suggestion.

[TeunVR]

I suggest to keep this issue open until the real app repos are available and then move the issue to the final app repos.
The current app repos are only for lab testing and not the final app.
Otherwise we keep making and closing issues, losing the discussion with it.

[dirkx]

Fair point.

[ijansch]

The term ‘reproducible builds’ is confusing. The thing the developers need to do to make their build reproducible is, if I understand correctly, to only publish which version of the source was used for a particular App Store build. The other parts in the linked examples above are the tasks that a verifier must perform to actually reproduce the builds. As such, these documents describe how to reproduce builds, but did not require changes to the codebase. Correct? I’m wondering if the instructions on how to reproduce the build should be documented as part of the source itself, since it includes the instruction to jailbreak a device, which can be considered harmful.

[TeunVR]

"Reproducible builds" is a common term used for the verification of pre-compiled binaries.
https://reproducible-builds.org

The idea is that anybody can build the same sources and get the same binary. The precompiled binary and self-built binary can then be compared (hash) to check wether the precompiled binary was in fact built with these sources.

The challenge in the case of iOS and Android is that it is hard or even impossible to get access to the binary from the AppStore / PlayStore and even if you get access the binary might be different (bitcode etc).

So the challenge is to either find a legal way to compare store-binaries or to implement a process where the builds are verified by independent parties before submitting to Apple/Google.

[arianvp]

Also see the technical documentation of the Italian colleagues.

https://github.com/immuni-app/immuni-documentation/blob/master/Technology%20Description.md#app-build-verification

Often; one should take extra caution when setting up your build system and CI pipeline to not introduce so-called impurities. Impurities include but are not limited to timestamps, non-determinism bugs in compilers, not defining what your system dependencies are precisely, etc.

if I understand correctly, to only publish which version of the source was used for a particular App Store build.

So this is often not enough. Attention has been taken from the start of development to apply reproducible builds, as not all source code or build pipelines produce reproducible binaries (that is; if you build the source code multiple times on multiple machines, the same artifact is generated)

They note they haven't been able to be successful yet in getting builds to be reproducible in iOS due to platform constraints , but the android builds are indeed reproducible to some extent.

They acknowledge that it's a complicated problem (it is!) and that's why they are focusing on having their CI pipelines as public as possible so that at least people can figure out "How" an artifact was built.

@ruuda note that F-Droid does not allow apps that depend on Google Play Services... so F-droid apps don't have access to the GACT APIs

[ijansch]

Great, thanks for this explanation. I will see how we can facilitate this. If somebody wants to assist, for example setting up the correct tooling, that would be very welcome. It looks like it takes a bit of effort and right now I’d like the team to be able to focus on getting the app ready.

[jellelicht]

@ijansch how do you see this happening exactly? Who/which group is currently responsible for creating the eventual releases?

[jellelicht]

Some teams have made some leeway with this here: https://github.com/DP-3T/dp3t-app-android-ch/blob/master/REPRODUCIBLE_BUILDS.md .

[jellelicht]

I think now is the time to make this happen (or not).
@ijansch very simple question: is there buy-in from all the relevant stakeholders on the internal side of things?

Sadly, this is one of the aspects that is very much only feasible to do once we know the exact build procedures used internally before pushing to Play store, so I think outsiders can't do much besides cheering and sharing relevant resources ;-).

EDIT: without this none of the privacy guarantees technically mean anything without some major reverse engineering efforts, so it seems an extremely bad PR move as well to not have these things in order long before the actual release.

[hvisser]

I know this won't be sufficient for what you're asking, but we're automating deploys to the Play Store and these build numbers relate to the build numbers on our CI. So it's not possible to upload a build with the same build number after the fact (Play Store won't allow this), nor is it possible to preempt an upload with a newer build number (our build will break).

[jellelicht]

@hvisser this is actually already a good step in the right direction; the most important piece of information left is sharing your build steps + build tools. For a typical CI system that usually boils down to the used container images, git checkout of the sources and the actual build instructions.

[hvisser]

Agreed, for Android it's pretty much one gradle command since the build tools etc are locked on a specific version already (tied to the AGP version) but we should document those steps. Recently we've added flavours to make it easier as well to reproduce a specific build.

[arianvp]

Gedurende de bouw zijn er voortdurend tests uitgevoerd naar de technische werking van de app. Om te bewijzen dat de versies van de apps die naar Google en Apple worden gestuurd 1-op-1 gebaseerd zijn op de versies die zijn gepubliceerd op GitHub, zijn door de Landsadvocaat in samenwerking met een zogenaamde “Escrow partij” (Escrow Alliance) validatiecontroles uitgevoerd. Voor elke versie van elke applicatie zal een verklaring van juistheid en integriteit worden afgegeven.

https://www.tweedekamer.nl/kamerstukken/brieven_regering/detail?id=2020Z14096&did=2020D29955

[jsoeterbroek]

With the merge of PR minvws/nl-covid19-notification-app-ios/pull/20, continuous integration testing & reproducible builds is now available to the iOS app as well, trough Github Actions workflow.

[arianvp]

I don't see how that PR solves the reproducible builds question. It allows us to build but there is no infrastructure to test whether the build is actually reproducible.

However; I think as OP suggested having an Escrow partner (and as also suggested as a solution to Parliament in the linked documented) is a good middle-ground that takes away many of the risks.

[jellelicht]

If an Escrow partner can do this, so can the community, right?
Why not both?

@jsoeterbroek is this Github Action the exact way that the release build of the application is made as well? Because then it is at least technically possible to have the App Store builds verified as well, and then the iOS part is done 🎆

(NB: Escrow for the server side of this all makes a lot of sense, so major 👍 for that)

[dirkx]

Yup - most certainly. We're working on cleanly triggering the builds; and hope to get to a stage where both are possible against same tag & comparable & publicly observable.

[jsoeterbroek]

I don't see how that PR solves the reproducible builds question. It allows us to build but there is no infrastructure to test whether the build is actually reproducible.

@arianvp: It's a (small) start towards this goal. I actually have no idea what is possible with signing and/or checksumming or other means.

[jsoeterbroek]

@jsoeterbroek is this Github Action the exact way that the release build of the application is made as well? Because then it is at least technically possible to have the App Store builds verified as well, and then the iOS part is done

@jellelicht: have no way of knowing. I am a volunteer contributer and I have no insight into how the internal release build of the application is made.

[jellelicht]

@jsoeterbroek that was a misunderstanding on my part then. When you mentioned reproducible builds being available, I'd assumed you meant something you didn't actually state.

[arianvp]

Do we agree we can close this now that escrow has been chosen as the path to tackle this risk?

[arianvp]

See https://github.com/minvws/nl-covid19-notification-app-provenance

Cc @dirkx

[dirkx]

Agreed. This can be closed.

[TeunVR]

Agreed. Thanks.

[dirkx]

Beetje jammer wel. Rep. builds waren mooier geweest. Volgende project :)

[hvisser]

We zijn heel dichtbij volgens mij, dus als je nog wat tijd overhebt @dirkx 😇