feat: macos signing and notarization #367
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,40 +2,287 @@ name: CI | |
|
|
||
| on: | ||
| push: | ||
| # TODO: restore before merging | ||
| # branches: | ||
| # - master | ||
| # pull_request: | ||
| # branches: | ||
| # - master | ||
|
|
||
| env: | ||
| DIST_ROOT: '/ipns/dist.ipfs.io' # content root used for calculating diff to build | ||
| GO_IPFS_VER: 'v0.9.1' # go-ipfs daemon used for chunking and applying diff | ||
| CLUSTER_CTL_VER: 'v0.14.0' # ipfs-cluster-ctl used for pinning | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: "ubuntu-latest" | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: actions/setup-node@v2 | ||
| with: | ||
| node-version: '14' | ||
| - name: Install ipfs and deps | ||
| run: | | ||
| curl -s https://dist.ipfs.io/go-ipfs/${GO_IPFS_VER}/go-ipfs_${GO_IPFS_VER}_linux-amd64.tar.gz | sudo tar vzx -C /usr/local/bin/ go-ipfs/ipfs --strip-components=1 | ||
| curl -s https://dist.ipfs.io/ipfs-cluster-ctl/${CLUSTER_CTL_VER}/ipfs-cluster-ctl_${CLUSTER_CTL_VER}_linux-amd64.tar.gz | sudo tar vzx -C /usr/local/bin/ ipfs-cluster-ctl/ipfs-cluster-ctl --strip-components=1 | ||
| - name: Fix resolv # DNS provided by Github is unreliable for DNSLik/dnsaddr | ||
| run: sudo sed -i -e 's/nameserver 127.0.0.*/nameserver 1.1.1.1/g' /etc/resolv.conf | ||
| - name: Set up ipfs | ||
| run: | | ||
| ipfs init --profile flatfs,server,test,lowpower | ||
| # make flatfs async for faster ci | ||
| new_config=$( jq '.Datastore.Spec.mounts[0].child.sync = false' ~/.ipfs/config) && echo "${new_config}" > ~/.ipfs/config | ||
| # restore deterministic port (changed by test profile) | ||
| ipfs config Addresses.API "/ip4/127.0.0.1/tcp/5001" | ||
| - name: Wait for ipfs daemon | ||
| run: ipfs daemon --enable-gc=false & while (! ipfs id --api "/ip4/127.0.0.1/tcp/5001"); do sleep 1; done | ||
| timeout-minutes: 3 | ||
| - name: Preconnect to cluster peers | ||
| run: | | ||
| echo '-> preconnect to cluster peers' | ||
| ipfs-cluster-ctl --enc=json \ | ||
| --host "/dnsaddr/ipfs-websites.collab.ipfscluster.io" \ | ||
| --basic-auth '${{ secrets.CLUSTER_USER }}:${{ secrets.CLUSTER_PASSWORD }}' \ | ||
| peers ls > cluster-peers-ls | ||
| for maddr in $(jq -r '.[].ipfs.addresses[]?' cluster-peers-ls); do | ||
| ipfs swarm connect "$maddr" || continue | ||
| done | ||
| echo '-> manual connect to cluster.ipfs.io' | ||
| ipfs swarm connect /dnsaddr/cluster.ipfs.io | ||
| echo '-> list swarm peers' | ||
| ipfs swarm peers | ||
| timeout-minutes: 3 | ||
| - name: Build any new ./releases | ||
| run: ./dockerized make all_dists | ||
| - name: Inspect git status and contents of ./releases | ||
| run: git status && ls -Rhl ./releases | ||
| - name: Temporarily save ./releases artifacts | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: releases-unsigned-diff | ||
| path: releases | ||
| retention-days: 1 | ||
|
|
||
| lint: | ||
| runs-on: "ubuntu-latest" | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: actions/setup-node@v2 | ||
| with: | ||
| node-version: '14' | ||
| - run: npm ci --no-audit --progress=false | ||
| - run: npm run lint | ||
|
|
||
| sign-macos: | ||
| runs-on: "macos-latest" | ||
| needs: build | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Retrieve unsigned artifacts | ||
| uses: actions/download-artifact@v2 | ||
| with: | ||
| name: releases-unsigned-diff | ||
| path: releases | ||
| continue-on-error: true # skip if no releases | ||
| - name: List ./releases before | ||
| run: ls -Rhl ./releases || echo "No ./releases" | ||
| - name: Install gon via HomeBrew for code signing and app notarization | ||
| run: | | ||
| brew tap mitchellh/gon | ||
| brew install ipfs coreutils gawk gnu-sed jq mitchellh/gon/gon | ||
|
Comment on lines
+72
to
+73
There was a problem hiding this comment. ℹ️ gon package is used by Terraform project, so multiple eyes are looking at this |
||
| ipfs init --profile test # needed for calculating NEW_CID later | ||
| - name: Import Keychain Certs | ||
| uses: apple-actions/import-codesign-certs@253ddeeac23f2bdad1646faac5c8c2832e800071 # v1@2020-02-03 | ||
|
There was a problem hiding this comment. ℹ️ 👮♂️ I hardcoded specific revision because this is a third-party action and in theory someone could swap-out git tag and steal our signing keys. |
||
| with: | ||
| p12-file-base64: ${{ secrets.APPLE_CERTS_P12 }} | ||
| p12-password: ${{ secrets.APPLE_CERTS_PASS }} | ||
| - name: Verify identity used for signing | ||
| run: security find-identity -v | ||
| - name: Unpack any new darwin arm64 and amd64 binaries to ./tmp | ||
| run: | | ||
| # ./releases/{DIST_NAME}/{DIST_VERSION}/*_darwin-${arch}.tar.gz -> ./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-unsigned/ | ||
| for NEW_DIR in ./releases/*/v*; do | ||
| (! test -d "$NEW_DIR") && continue | ||
| DIST_VERSION=$(basename "$NEW_DIR") | ||
| DIST_NAME=$(basename $(dirname "$NEW_DIR")) | ||
| for arch in "amd64" "arm64"; do | ||
| mkdir -p "./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-unsigned" | ||
| tar -zxvf "./releases/${DIST_NAME}/${DIST_VERSION}/${DIST_NAME}_${DIST_VERSION}_darwin-${arch}.tar.gz" -C "./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-unsigned/" | ||
| done | ||
| done | ||
| ls -Rhl ./tmp || echo "Nothing new in ./tmp" | ||
| - name: Sign and notarize the mac binaries | ||
| env: | ||
| AC_USERNAME: ${{ secrets.APPLE_AC_USERNAME }} # implicitly read from env by gon | ||
| AC_PASSWORD: ${{ secrets.APPLE_AC_PASSWORD }} | ||
| run: | | ||
| # Find and sign executables in ./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-unsigned/ | ||
| for NEW_DIR in ./releases/*/v*; do | ||
| (! test -d "$NEW_DIR") && continue | ||
| DIST_VERSION=$(basename "$NEW_DIR") | ||
| DIST_NAME=$(basename $(dirname "$NEW_DIR")) | ||
| for arch in "amd64" "arm64"; do | ||
| EXECUTABLES=$(jq -nc '$ARGS.positional' --args $(find "./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-unsigned/" -perm +111 -type f -print)) | ||
| echo "{ | ||
| \"source\" : $EXECUTABLES, | ||
| \"bundle_id\" : \"io.ipfs.dist.${DIST_NAME}\", | ||
| \"apple_id\": { | ||
| \"password\": \"@env:AC_PASSWORD\" | ||
| }, | ||
| \"sign\" :{ | ||
| \"application_identity\" : \"Developer ID Application: Protocol Labs, Inc. (7Y229E2YRL)\" | ||
| }, | ||
| \"zip\" :{ | ||
| \"output_path\" : \"./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-signed.zip\" | ||
| } | ||
| }" | tee | jq > "./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-gon.json" | ||
| gon -log-level=info -log-json "./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-gon.json" | ||
| done | ||
| done | ||
| - name: Temporarily save ./tmp | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: tmp | ||
| path: ./tmp/ | ||
| retention-days: 1 | ||
| - name: Update changed binaries in ./releases | ||
| run: | | ||
| for NEW_DIR in ./releases/*/v*; do | ||
| (! test -d "$NEW_DIR") && continue | ||
| DIST_VERSION=$(basename "$NEW_DIR") | ||
| DIST_NAME=$(basename $(dirname "$NEW_DIR")) | ||
| for arch in "amd64" "arm64"; do | ||
lidel marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| echo "-> Starting the update of darwin_${arch}.tar.gz for name='${DIST_NAME}' and version='${DIST_VERSION}'" | ||
| # unzip signed binaries to a directory matching .tar.gz structure | ||
| cd "${{ github.workspace }}" | ||
| mkdir -p "./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-signed/${DIST_NAME}" | ||
| cd "./tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-signed/${DIST_NAME}/" | ||
| echo "-> Unpacking gon .zip for ${arch}" | ||
| unzip "${{ github.workspace }}/tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-signed.zip" | ||
| echo "-> Unpacked contents" | ||
| ls -Rhl "${{ github.workspace }}/tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-signed/" | ||
| # replace .tar.gz with one that has the same structure, but signed binaries | ||
| PKG_NAME="${DIST_NAME}_${DIST_VERSION}_darwin-${arch}.tar.gz" | ||
| PKG_ROOT="${{ github.workspace }}/releases/${DIST_NAME}/${DIST_VERSION}" | ||
| PKG_PATH="${PKG_ROOT}/${PKG_NAME}" | ||
| DIST_JSON="${PKG_ROOT}/dist.json" | ||
| # read old hashes | ||
| OLD_CID=$(cat "${PKG_PATH}.cid") | ||
| OLD_SHA512=$(gawk '{ print $1; }' < "${PKG_PATH}.sha512") | ||
| echo "-> Found old $PKG_NAME" | ||
| echo " old CID: $OLD_CID" | ||
| echo " old SHA512: $OLD_SHA512" | ||
| echo "-> Updating $PKG_NAME" | ||
| rm "$PKG_PATH" | ||
| tar -czvf "${{ github.workspace }}/releases/${DIST_NAME}/${DIST_VERSION}/$PKG_NAME" -C "${{ github.workspace }}/tmp/${DIST_NAME}_${DIST_VERSION}_${arch}-signed/" "${DIST_NAME}" | ||
| # calculate new hashes | ||
| NEW_CID=$(ipfs add -Qn "$PKG_PATH") | ||
| NEW_SHA512_LINE=$(gsha512sum "$PKG_PATH") | ||
| NEW_SHA512=$(echo "$NEW_SHA512_LINE" | gawk '{ print $1; }') | ||
| echo "-> New $PKG_NAME" | ||
| echo " new CID: $NEW_CID" | ||
| echo " new SHA512: $NEW_SHA512" | ||
| # update metadata to use new hashes | ||
| echo "$NEW_CID" > "${PKG_PATH}.cid" | ||
| echo "$NEW_SHA512_LINE" > "${PKG_PATH}.sha512" | ||
| gsed -i "s/${OLD_CID}/${NEW_CID}/g; s/${OLD_SHA512}/${NEW_SHA512}/g" "${PKG_ROOT}/dist.json" | ||
| echo "-> Completed the update of ${arch}.tar.gz for ${DIST_NAME} ${DIST_VERSION}" | ||
| done | ||
| done | ||
| - name: List ./releases after | ||
| run: ls -Rhl ./releases || echo "No ./releases" | ||
| - name: Temporarily save notarized artifacts | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: releases-signed-macos-diff | ||
| path: releases | ||
| retention-days: 1 | ||
| continue-on-error: true # skip if no releases | ||
|
|
||
| persist: | ||
| runs-on: "ubuntu-latest" | ||
| needs: sign-macos | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Retrieve signed artifacts | ||
| uses: actions/download-artifact@v2 | ||
| continue-on-error: true # skip if no releases | ||
| with: | ||
| name: releases-signed-macos-diff | ||
| path: releases | ||
| - name: List ./releases | ||
| run: ls -Rhl ./releases || echo "No ./releases" | ||
| - name: Install ipfs and deps | ||
| run: | | ||
| curl -s https://dist.ipfs.io/go-ipfs/${GO_IPFS_VER}/go-ipfs_${GO_IPFS_VER}_linux-amd64.tar.gz | sudo tar vzx -C /usr/local/bin/ go-ipfs/ipfs --strip-components=1 | ||
| curl -s https://dist.ipfs.io/ipfs-cluster-ctl/${CLUSTER_CTL_VER}/ipfs-cluster-ctl_${CLUSTER_CTL_VER}_linux-amd64.tar.gz | sudo tar vzx -C /usr/local/bin/ ipfs-cluster-ctl/ipfs-cluster-ctl --strip-components=1 | ||
| - name: Fix resolv # DNS provided by Github is unreliable for DNSLik/dnsaddr | ||
| run: sudo sed -i -e 's/nameserver 127.0.0.*/nameserver 1.1.1.1/g' /etc/resolv.conf | ||
| - name: Set up ipfs | ||
| run: | | ||
| ipfs init --profile flatfs,server,test,lowpower | ||
| # make flatfs async for faster ci | ||
| new_config=$( jq '.Datastore.Spec.mounts[0].child.sync = false' ~/.ipfs/config) && echo "${new_config}" > ~/.ipfs/config | ||
| # restore deterministic port (changed by test profile) | ||
| ipfs config Addresses.API "/ip4/127.0.0.1/tcp/5001" | ||
| - name: Wait for ipfs daemon | ||
| run: ipfs daemon --enable-gc=false & while (! ipfs id --api "/ip4/127.0.0.1/tcp/5001"); do sleep 1; done | ||
| timeout-minutes: 3 | ||
| - name: Preconnect to cluster peers | ||
| run: | | ||
| echo '-> preconnect to cluster peers' | ||
| ipfs-cluster-ctl --enc=json \ | ||
| --host "/dnsaddr/ipfs-websites.collab.ipfscluster.io" \ | ||
| --basic-auth '${{ secrets.CLUSTER_USER }}:${{ secrets.CLUSTER_PASSWORD }}' \ | ||
| peers ls > cluster-peers-ls | ||
| for maddr in $(jq -r '.[].ipfs.addresses[]?' cluster-peers-ls); do | ||
| ipfs swarm connect "$maddr" || continue | ||
| done | ||
| echo '-> manual connect to cluster.ipfs.io' | ||
| ipfs swarm connect /dnsaddr/cluster.ipfs.io | ||
| echo '-> list swarm peers' | ||
| ipfs swarm peers | ||
| timeout-minutes: 3 | ||
| - run: ./dockerized make publish | ||
| - run: git status | ||
| - name: Read CID of updated DAG | ||
| id: cid-reader | ||
| run: echo "::set-output name=CID::$(tail -1 ./versions)" | ||
| - name: Pin new website to ipfs-websites.collab.ipfscluster.io | ||
| run: | | ||
| echo 'pin to cluster' | ||
| ipfs-cluster-ctl --enc=json \ | ||
| --host "/dnsaddr/ipfs-websites.collab.ipfscluster.io" \ | ||
| --basic-auth '${{ secrets.CLUSTER_USER }}:${{ secrets.CLUSTER_PASSWORD }}' \ | ||
| pin add \ | ||
| --pin-name="https://github.com/ipfs/distributions/commits/${{ github.sha }}" \ | ||
| --no-status \ | ||
| "${{ steps.cid-reader.outputs.CID }}" | ||
| while true; do | ||
| ipfs-cluster-ctl --enc=json \ | ||
| --host "/dnsaddr/ipfs-websites.collab.ipfscluster.io" \ | ||
| --basic-auth '${{ secrets.CLUSTER_USER }}:${{ secrets.CLUSTER_PASSWORD }}' \ | ||
| status "${{ steps.cid-reader.outputs.CID }}" | tee cluster-pin-status | ||
| if [[ $(jq '.peer_map[].status' cluster-pin-status | grep '"pinned"' | wc -l) -ge 2 ]]; then | ||
| echo "Got 2 pin confirmations, finishing the workflow" | ||
| break | ||
| else | ||
| echo "(sleeping for 15 seconds)" | ||
| sleep 15 | ||
| fi | ||
| done | ||
| timeout-minutes: 60 | ||
| - name: Update PR status with preview link | ||
| run: | | ||
| PREVIEW_URL="https://dweb.link/ipfs/${{ steps.cid-reader.outputs.CID }}" | ||
| API_PARAMS=$(jq --monochrome-output --null-input \ | ||
| --arg state "success" \ | ||
| --arg target_url "$PREVIEW_URL" \ | ||
| --arg description "Preview on IPFS" \ | ||
| --arg context "IPFS" \ | ||
| '{ state: $state, target_url: $target_url, description: $description, context: $context }' ) | ||
| curl --output /dev/null --silent --show-error \ | ||
| -X POST -H 'Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' -H 'Content-Type: application/json' \ | ||
| --data "$API_PARAMS" 'https://api.github.com/repos/ipfs/distributions/statuses/${{ github.sha }}' | ||
| echo "Pinned to IPFS - $PREVIEW_URL" | ||
|
There was a problem hiding this comment. This mechanism for adding a status result to a PRs checks list stopped working for me in ipfs-dns-deploy. I tried to fix it, but could never get the status to appear again on the PR... what did you change to make it work here!? There was a problem hiding this comment. @olizilla Hm.. not sure, could be one of below:
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,4 +25,4 @@ if [ "$failed" = true ]; then | |
| fi | ||
|
|
||
| echo "npm install" | ||
| exec npm ci --prefer-offline --no-audit --progress=false | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the process for updating these? Do we rely on someone manually updating it? What's their trigger for doing so?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think those are low importance: could be kept as-is until someone cares to update or there is a breaking change that forces version bump.
Having static versions of build tools makes the builds more robust and reproducible, but we could also switch to always run on the latest version for dogfooding – don't feel strongly either way, but leaning towards more reproducibility.