Using Nova Reusable Build Workflows

Creating your own pipelines for building and distributing binaries is hard and messy. Monitoring, alerting, and verification are even harder. Let us help you! With Nova Reusable Workflows, you have a strong set of shared components that you can simply call to get binary builds and distribution working across your entire build matrix out-of-the-box. For more involved build processes, we have a number of hooks for domain-specific customizability. No need to copy dozens of bash scripts between repos any longer!

Quick Start

Let's say you want to distribute Python Wheels for Linux systems for your package. Create a file called .github/workflows/build-wheels-linux.yml in your repo. Add the following to the file, and we'll walk through the changes you must make:

name: Build Linux Wheels

on:
  pull_request:
  push:
    branches:
      - nightly
  workflow_dispatch:

jobs:
  generate-matrix:
    uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main
    with:
      package-type: wheel
      os: linux
      test-infra-repository: pytorch/test-infra
      test-infra-ref: main
      with-cuda: disable
  build:
    needs: generate-matrix
    strategy:
      fail-fast: false
      matrix:
        include:
            # TODO #1: Enter the name of your repo
          - repository: <Your_Organization_Name/Your_Repo_Name>
            # TODO #2: Have custom build steps before building the wheel? Pass the path to the shell script with those steps below.
            pre-script: <Your_Custom_Pre_Build_Script>
            # TODO #3: Same as above, but post-build steps. If you have none, just use "".
            post-script: <Your_Custom_Post_Build_Script>
            # TODO #4: Want to verify the correctness of your binaries before distributing them? Add custom python smoke tests below.
            smoke-test-script: <Your_Smoke_Tests>
            # TODO #5: Enter the name of your python package below. (for example, torchvision)
            package-name: torchtext
    name: ${{ matrix.repository }}
    uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main
    with:
      repository: ${{ matrix.repository }}
      ref: ""
      test-infra-repository: pytorch/test-infra
      test-infra-ref: main
      build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
      pre-script: ${{ matrix.pre-script }}
      post-script: ${{ matrix.post-script }}
      package-name: ${{ matrix.package-name }}
      smoke-test-script: ${{ matrix.smoke-test-script }}
      trigger-event: ${{ github.event_name }}
    secrets:
      AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID: ${{ secrets.AWS_PYTORCH_UPLOADER_ACCESS_KEY_ID }}
      AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY: ${{ secrets.AWS_PYTORCH_UPLOADER_SECRET_ACCESS_KEY }}

And that's it! Every time a push is done to your repo's nightly branch, this will build, verify, and upload nightlies to download.pytorch.org. Every time a push is done to your release candidate branches, the same workflow will be run. And official releases will also be done via this workflow.

How It Works

We have 6 "base" workflows in pytorch/test-infra that take care of most of the logic for creating and uploading binaries. The workflows cover the set {Linux, MacOS, Windows} x {Wheels, Conda}:

• build_wheels_linux.yml
• build_wheels_macos.yml
• build_wheels_windows.yml
• build_conda_linux.yml
• build_conda_macos.yml
• build_conda_windows.yml

To run these workflows, you need to write a "caller" workflow in your repo. It is best to have a separate caller workflow for each base workflow you want to call. Each caller workflow does 3 things:

1. Configure Triggers (when should build workflow should be run?)
2. Generate the Build Matrix
3. Calls the base workflow

Configure Triggers

The fields you define in the on: section of the yaml config determine which triggers your workflow runs after. This part is vanilla GitHub Actions, so this should be a good reference: Triggering a Workflow. Generally you want your workflows to run on pushes to nightly and release candidate branches, so you can list those branches under the push: field. Additionally, you may want these jobs triggered on PR's, so add the pull_request: field.

Generate the Build Matrix

The pytorch/test-infra repo contains a workflow that generates a build matrix based on provided inputs. For example, you'll typically want to support various Python Versions, GPU architectures, etc. This build matrix enumerates all of these configurations that we will build your binaries with. There are 3 key inputs you provide the build matrix generation job: os, package-type, and with-cuda. The os is the Operating System you want these binaries to support. The package-type is either wheel or conda (the package you are building). with-cuda toggles whether you need to build separate binaries with GPU support. If not, you can set with-cuda to disable.

Calling the base workflow