Fix numpy ABI compatible and Use CIBuildWheel and github action to bu…

…ild python wheels automatically (#67) * use oldest-supported-numpy; adjust cypthon install requirement setup * add cibuildwheel in github action * change the numpy range to 1.14.5, according to the oldest_support_numpy https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg\#L54 * change numpy version in setup.py * test override numpy version for python 3.7 * update matrix os variables * remove unnecessary setup requirement in setup.py * fix typo? * add 3.6 * workaround for error in windows os: CCompiler_spawn() got an unexpected keyword argument https://github.com/pypa/distutils/issues/15\#issuecomment-685029840 * skip pypy 3.7 macos, due to error would build wheel with unsupported tag pp37 pypy37_pp73 macosx_10_9_x86_64 * add pytest after build * add pytest folder * add CIBW archs * indicate tests folder explicitly * fix test in CIBW * skip pypy37 for mac linux and window * also skip other pypy for mac linux and window * skip all pypy for mac linux and window * test cibuildwheel 2.6.1, since 2.7 gives standard_init_linux.go:228: exec user process caused: exec format error * try use python -m to see if can workaroudn the aarch64 error * use different pytest command * skip linux aarch64 * change version name to 0.3.2rc3 * fix musllinux No lapack/blas resources found * skip musllinux * add pypi into release destination * update version and update release strategy * update version and readme Co-authored-by: Zhe Sun <info@sun-analytics.nl> Co-authored-by: Stephane Collot <stephane.collot@ing.com>
ing-bank · Jun 27, 2022 · a6f8d06 · a6f8d06
1 parent 09d1ec8
commit a6f8d06
Show file tree

Hide file tree

Showing 6 changed files with 122 additions and 15 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -10,7 +10,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        python-version: [3.7, 3.8, 3.9]
+        python-version: [3.6, 3.7, 3.8, 3.9]
         os: [ubuntu-latest, windows-latest]
 
     steps:

diff --git a/.github/workflows/wheels.yml b/.github/workflows/wheels.yml
@@ -0,0 +1,98 @@
+name: Build Python wheels and publish
+
+# Controls when the workflow will run
+on:
+  # run pipeline on push event of main or release branch
+  push:
+    branches:
+      - 'test/**'
+      - 'release/**'
+  # run pipeline on pull request
+  pull_request:
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  build-and-test-python:
+    strategy:
+      matrix:
+        platform: [linux, macos, windows]
+        include:
+          - platform: linux
+            os: ubuntu-latest
+            # Here we skip aarch64, since we got error
+            # ` standard_init_linux.go:228: exec user process caused: exec format error`
+            # and we don't know how to fix it
+            archs: "x86_64" 
+          - platform: macos
+            os: macos-latest
+            archs: "x86_64 arm64"
+          - platform: windows
+            os: windows-latest
+            archs: AMD64
+
+    runs-on: ${{ matrix.os }}
+
+    env:
+      CIBW_ENVIRONMENT_WINDOWS: SETUPTOOLS_USE_DISTUTILS=stdlib
+      CIBW_SKIP: "pp3* *-musllinux_*"
+      CIBW_ARCHS: ${{ matrix.archs }}
+      CIBW_TEST_REQUIRES: pytest pandas
+      CIBW_TEST_COMMAND: pytest -ra --capture=no --showlocals {package}/tests
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Build wheels
+        uses: pypa/cibuildwheel@2.6.1
+
+      - name: Keep wheel files
+        uses: actions/upload-artifact@v3
+        with:
+          name: wheelhouse
+          path: ./wheelhouse/*.whl
+
+
+  publish-wheels:
+    needs: build-and-test-python
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Setup python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+          architecture: x64
+
+      - uses: actions/download-artifact@v3
+        with:
+          name: wheelhouse
+          path: wheelhouse
+
+      - name: List assets
+        run: |
+          ls ./wheelhouse/*.whl -al
+
+      - name: Upload wheels to test PyPI
+        if: (github.event_name == 'push') || (github.event_name == 'workflow_dispatch')
+        run: |
+          pip install twine
+          echo "Publish to Test PyPI..."
+          twine upload --verbose --repository testpypi wheelhouse/*
+        env:
+          TWINE_USERNAME: ${{ secrets.PYPI_TEST_USER }}
+          TWINE_PASSWORD: ${{ secrets.PYPI_TEST_PASS }}
+
+      - name: Upload wheels to official PyPI
+        if: contains(github.ref, 'release') && ((github.event_name == 'push') || (github.event_name == 'workflow_dispatch'))
+        run: |
+          pip install twine
+          echo "Publish to PyPI..."
+          twine upload --verbose wheelhouse/*
+        env:
+          TWINE_USERNAME: ${{ secrets.PYPI_USER }}
+          TWINE_PASSWORD: ${{ secrets.PYPI_PASS }}
diff --git a/README.md b/README.md
@@ -64,3 +64,19 @@ python -m build
 cd dist/
 pip install sparse_dot_topn-*.tar.gz
 ```
+
+## Release strategy
+From version 0.3.2, we employ Github Actions to build wheels in different OS environment and release automatically. Hopefully this will solve many issues related to installation. The build and publish pipeline is configured in `./github/workflows/wheels.yml`. When a new release is neeeded, please follow these steps
+
+1. Create a test branch with branch name `test/x.x.x` from main branch.
+2. In `test/x.x.x` branch, update the version number such as `x.x.x.rcx`in setup.py, and update changelog in CHANGES.md file. 
+3. Git push `test/x.x.x` branch, then build and publish pipeline will be triggered automatically. New release will be uploaded in PyPI test [https://test.pypi.org/project/sparse-dot-topn/](https://test.pypi.org/project/sparse-dot-topn/).
+4. Please do a sanity check on PyPI test release.
+5. Create a branch on top of the test branch.
+6. Modify the version number by remove the `rcx` surfix.
+7. Git push, then build and publish pipeline will be triggered automatically. New release will be uploaded to PyPI [https://pypi.org/project/sparse-dot-topn](https://pypi.org/project/sparse-dot-topn/)
+8. Merge the release branch back to master
+
+
+
+
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,8 @@
 [build-system]
 requires = [
     "setuptools>=42",
-    "wheel"
+    "wheel",
+    "cython",
+    "oldest-supported-numpy"
 ]
 build-backend = "setuptools.build_meta"
diff --git a/requirements.txt b/requirements.txt
@@ -1,4 +1,4 @@
-numpy>=1.16.6
+numpy>=1.14.5
 setuptools>=45.2.0
 Cython>=0.29.15
 scipy>=1.4.1
diff --git a/setup.py b/setup.py
@@ -64,7 +64,7 @@ def _set_builtin(name, value):
 
 setup(
     name='sparse_dot_topn',
-    version='0.3.1',
+    version='0.3.2',
     description='This package boosts a sparse matrix multiplication '\
                 'followed by selecting the top-n multiplication',
     keywords='cosine-similarity sparse-matrix scipy cython',
@@ -74,18 +74,9 @@ def _set_builtin(name, value):
     author='Zhe Sun',
     author_email='ymwdalex@gmail.com',
     license='Apache 2.0',
-    setup_requires=[
-        # Setuptools 18.0 properly handles Cython extensions.
-        'setuptools>=42',
-        'cython>=0.29.15',
-        'numpy>=1.16.6', # select this version for Py2/3 compatible
-        'scipy>=1.2.3'   # select this version for Py2/3 compatible
-    ],
     install_requires=[
-        # Setuptools 18.0 properly handles Cython extensions.
-        'setuptools>=42',
-        'cython>=0.29.15',
-        'numpy>=1.16.6', # select this version for Py2/3 compatible
+        # select this version due to oldest_support_numpy https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg#L54
+        'numpy>=1.14.5',
         'scipy>=1.2.3'   # select this version for Py2/3 compatible
     ],
     zip_safe=False,