Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: drop 3.6 support #1661

Merged
merged 18 commits into from
Sep 3, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 6 additions & 13 deletions .github/workflows/build-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ jobs:
- '3.9'
- '3.8'
- '3.7'
- '3.6'

python-architecture:
- x64
Expand Down Expand Up @@ -91,7 +90,6 @@ jobs:
- '3.9'
- '3.8'
- '3.7'
- '3.6'

runs-on: macOS-11

Expand Down Expand Up @@ -148,11 +146,11 @@ jobs:
- '3.9'
- '3.8'
- '3.7'
- '3.6'

numpy-package:
- "numpy"
include:
- python-version: '3.6'
numpy-version: 1.13.1
- python-version: '3.7'
numpy-package: "oldest-supported-numpy"

runs-on: ubuntu-20.04

Expand All @@ -170,7 +168,7 @@ jobs:
with:
key: >-
${{ github.job
}}-${{matrix.python-version}}-${{matrix.numpy-version}}
}}-${{matrix.python-version}}-${{matrix.numpy-package}}

- name: Use ccache
run: |
Expand All @@ -183,12 +181,7 @@ jobs:
python-version: '${{ matrix.python-version }}'

- name: Install NumPy
run: |
if [[ -z "${NUMPY_VERSION}" ]]; then
python -m pip install numpy;
else
python -m pip install "numpy==${NUMPY_VERSION}";
fi
run: python -m pip install "${{ matrix.numpy-package }}"

- name: Build
run: 'python -m pip install -v .[test,dev]'
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,7 @@ jobs:
- uses: pypa/cibuildwheel@v2.9.0
env:
CIBW_ARCHS_MACOS: universal2
CIBW_MANYLINUX_X86_64_IMAGE: manylinux1
CIBW_BUILD: cp39-win_amd64 cp36-manylinux_x86_64 cp38-macosx_universal2
CIBW_BUILD: cp39-win_amd64 cp37-manylinux_x86_64 cp38-macosx_universal2

- uses: pypa/cibuildwheel@v2.9.0
if: matrix.os == 'ubuntu-latest'
Expand Down
11 changes: 2 additions & 9 deletions .github/workflows/wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,6 @@ jobs:
build: ["cp*", "pp*"]

include:
- os: ubuntu-latest
type: ManyLinux1
arch: auto64
build: "cp{36,37,38}-*"
CIBW_MANYLINUX_X86_64_IMAGE: manylinux1

- os: macos-latest
type: "Universal"
arch: universal2
Expand All @@ -51,7 +45,7 @@ jobs:

- os: windows-latest
arch: auto32
build: "cp{36,37,38,39}-*"
build: "cp{37,38,39}-*"


steps:
Expand All @@ -62,7 +56,6 @@ jobs:
- uses: pypa/cibuildwheel@v2.9.0
env:
CIBW_BUILD: ${{ matrix.build }}
CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.CIBW_MANYLINUX_X86_64_IMAGE }}
CIBW_ARCHS: ${{ matrix.arch }}

- name: Upload wheels
Expand All @@ -76,7 +69,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python: [36, 37, 38, 39, 310, 311]
python: [37, 38, 39, 310, 311]
arch: [aarch64]
steps:

Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ repos:
rev: "v2.37.3"
hooks:
- id: pyupgrade
args: ["--py36-plus"]
args: ["--py37-plus"]

- repo: https://github.com/psf/black
rev: 22.6.0
Expand Down
87 changes: 26 additions & 61 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,20 @@ That said, most of the commits on a pull request/git branch should be from a sin

As such, you should name your branch starting with your GitHub userid and a slash, such as `jpivarski/write-contributing-md`. If you start a pull request with a branch that doesn't follow convention, though, you don't need to fix it.

The titles of pull requests (and therefore the merge commit messages) should follow [these conventions](https://www.conventionalcommits.org/en/v1.0.0/#summary). Mostly, this means prefixing the title with one of these words and a colon:

* feat: new feature
* fix: bug-fix
* perf: code change that improves performance
* refactor: code change that neither fixes a bug nor adds a feature
* style: changes that do not affect the meaning of the code
* test: adding missing tests or correcting existing tests
* build: changes that affect the build system or external dependencies
* docs: documentation only changes
* ci: changes to our CI configuration files and scripts
* chore: other changes that don't modify src or test files
* revert: reverts a previous commit

Most pull requests are merged with the "squash and merge" feature, so details about commit history within a pull request are lost. Feel free, therefore, to commit with any frequency you're comfortable with. I like to make frequent commits to avoid losing work to a dead laptop, and to have more save-points to recover from.

It is unnecessary to manually edit (rebase) your commit history. If, however, you do want to save a pull request as multiple commits on `main`, ask me and we'll discuss. (The default branch is named `main`, not `master`.)
Expand All @@ -55,15 +69,7 @@ python -m pytest -vv -rs tests-spec
python -m pytest -vv -rs tests-cpu-kernels
```

Furthermore, if you have an Nvidia GPU, you can build and locally install the experimental CUDA plug-in with

```bash
pip uninstall -y awkward-cuda-kernels
python dev/generate-cuda.py
./cuda-build.sh --install
```

The `--install` does a local `pip install` on your system, which is the only way to use it. You can run its tests with
Furthermore, if you have an Nvidia GPU and CuPy installed, you can run the CUDA tests with

```bash
python dev/generate-tests.py
Expand Down Expand Up @@ -114,45 +120,12 @@ Before re-building documentation, you might want to delete the files that were g
rm -rf docs-shinx/_auto docs_sphinx/_build docs-sphinx/_static
```

### Continuous integration
### Continuous testing

Pull requests must pass all [continuous integration](https://dev.azure.com/jpivarski/Scikit-HEP/_build?definitionId=3&_a=summary) tests before they are merged. I will sometimes cancel non-essential builds to give priority to pull requests that are almost ready to be merged. If you needed the result of the build as a diagnostic, you can ask me to restart your job or make a trivial change to trigger a new build.

Currently, we only run merge builds (the state of your branch if merged with `main`), not regular branch builds (the state of your branch as-is), because only merge builds can be made to run for pull requests from external forks and it makes better use of our limited execution time on Azure. If you want to enable regular branch builds, you can turn it on for your branch by editing `trigger/branches/exclude` in [.ci/azure-buildtest-awkwrad.yml](https://github.com/scikit-hep/awkward-1.0/blob/9b6fca3f6e6456860ae40979171f762e0045ce7c/.ci/azure-buildtest-awkward.yml#L1-L5). The merge build trigger is not controlled by the YAML file. It is better, however, to keep up-to-date with `git merge main`.

### Semi-automated testing of CUDA kernels

For development on the cuda-kernels, an AWS VM with a GPU has been set up to run tests in the `tests-cuda` directory. You can email jpivarski at GMail for more information on how to get permissions to launch the AWS instance.

To see if the AWS instance is running, use the launcher:

```bash
% ssh -i ~/.ssh/awkward1-cuda-test.pem ubuntu@WW.XX.YY.ZZ ./list-instances.py
i-0295dd31185ce78f7 WW.XX.YY.ZZ running ['launcher']
i-0a1ec83c261607797 AA.BB.CC.DD running ['awkward1-cuda-test']
i-0a9dd553fe19610fb (none) stopped ['awkward1-cuda-test-2gpus']
```

If it is `stopped`, launch it with:

```bash
ssh -i ~/.ssh/awkward1-cuda-test.pem ubuntu@WW.XX.YY.ZZ aws ec2 start-instances --instance-ids i-0a1ec83c261607797
```

Once it's `running`, use its IP address (`AA.BB.CC.DD`) to run the tests for your branch (`my-branch-name`):

```bash
ssh -i ~/.ssh/awkward1-cuda-test.pem ubuntu@AA.BB.CC.DD ./cuda-run-tests.sh my-branch-name
```

When you are done, be sure to put it back into the `stopped` state:

```bash
ssh -i ~/.ssh/awkward1-cuda-test.pem ubuntu@WW.XX.YY.ZZ aws ec2 stop-instances --instance-ids i-0a1ec83c261607797
```

and verify with `list-instances`. While the GPU instance is running, you can ssh into it interactively for debugging.

### The main branch

The Awkward Array `main` branch must be kept in an unbroken state. Although the recommended way to install Awkward Array is through pip or conda, the `main` branch on GitHub must always be functional. Pull requests for bug fixes and new features are based on `main`, so it has to work for users to test our proposed changes.
Expand All @@ -167,6 +140,10 @@ Committing directly to `main` is not allowed except for

and only by me.

### The main-v1 branch

The `main-v1` branch was split from `main` just before Awkward 1.x code was removed, so it exists to make 1.10.x bug-fix releases. These commits must be be drawn from `main-v1`, not `main`, and pull requests must target `main-v1` (not the GitHub default). A single commit cannot be applied to both `main` and `main-v1` because they have diverged too much. If a bug-fix needs to be applied to both (unlikely), it will have to be reimplemented on both.

### Releases

Similarly, only I publish releases (or a team of maintainers, in the future). Publishing releases starts the deployment procedure, updating the package that users will get when they pip-install.
Expand All @@ -190,16 +167,6 @@ Contributing to each part of the codebase has a different flavor:

A Contribution might only touch one layer of the code or it might involve more than one.

### Deployment products

Awkward Array consists of two (or three) shared libraries,

* `libawkward-cpu-kernels.so` (or `.dylib`, etc.)
* `libawkward-cuda-kernels.so` (if compiled with CUDA support)
* `libawkward.so`

and a Python library shipped as binary wheels or (in rare cases) a source tarball to be compiled. The pip and conda deployments include all shared libraries with the Python package and put C++ headers in the system-dependent location for `includes`, in a directory named `awkward`.

### Performance considerations

The conventional model is that Python is for a good user interface and C++ is for performance. In the case of Awkward Array, even the C++ layer is not intended for high performance; this is pushed down to the CPU and GPU kernels. In typical applications, the number of C++ objects is small (hundreds to thousands of instances) while the size of array buffers sent to CPU and GPU kernels is large (billions of elements).
Expand Down Expand Up @@ -259,19 +226,17 @@ As stated above, most of the C++ features we use are "old," such as virtual inhe

A quick scan of the code would reveal that we make extensive use of `std::shared_ptr`. Raw pointers are used in extremely limited circumstances, and never passed out of the scope that holds a shared pointer to the same resource (guaranteeing its lifetime). The main use of raw pointers is for down-casting with `dynamic_cast`.

Class instances are passed between functions as generic types for flexibility. For example, `toListOffsetArray64` returns a list array converted to the `ListOffsetArray64` class, but it is passed as a smart pointer to the common superclass, `Content`. This is because passing as a `ListOffsetArray64` exposes typeinfo symbols that violate visibility rules.

Class hierarchies are intentionally simple. If any inheritance is involved, the one superclass is abstract and all subclasses are direct children of that abstract superclass.

Templating is only used for integer specialization.
Templating is mostly just used for integer specialization.

### Python standard

We target Python 3.6 and above. Import statements can assume Python 3 names, string-checking can assume Python 3 meanings of `str` and `bytes`, Unicode literals don't need to be prefixed by `u`, and dict order can be assumed to be stable. Python's f-strings can now be used, but not with equals signs (e.g. `f"{something = }"` rather than `f"something = {something}"`) because that's a Python 3.8 feature (its main use is in debugging, anyway).
We target Python 3.7 and above. Import statements can assume Python 3 names, string-checking can assume Python 3 meanings of `str` and `bytes`, Unicode literals don't need to be prefixed by `u`, and dict order can be assumed to be stable. Python's f-strings can now be used, but not with equals signs (e.g. `f"{something = }"` rather than `f"something = {something}"`) because that's a Python 3.8 feature (its main use is in debugging, anyway).

If you see any outdated (pre-Python 3.6) code, you can safely clean them up. Some strings are not easier to read as f-strings or require some work to make them more readable; it should be a case-by-case basis.
If you see any outdated (pre-Python 3.7) code, you can safely clean them up. Some strings are not easier to read as f-strings or require some work to make them more readable; it should be a case-by-case basis.

Python 3.6 will be the standard until it becomes "painful" to keep it (like, when CI and build tools no longer support it, or if we adopt type annotations).
Awkward Array follows [CPython's EoL schedule](https://endoflife.date/python), and will drop support for out-of-date versions of Python accordingly.

### Third party dependencies

Expand All @@ -281,11 +246,11 @@ The Python codebase only strictly depends on NumPy 1.13.1, the first version wit

Other third party libraries are used if they exist (can be imported), and we only accept certain versions of these libraries. Both the test-import and any version-testing must be within runtime code, not startup code, so that they're only invoked when users explicitly call for the feature that requires them.

Versions can be explicitly tested with `distutils.version.LooseVersion`, though it's better to test for features (existence of classes, methods, and attributes) than to test for explicit version numbers.
Versions can be explicitly tested with `packaging.version.parse`, though it's better to test for features (existence of classes, methods, and attributes) than to test for explicit version numbers.

### Array object details

Arrays are (or soon will be: [#176](https://github.com/scikit-hep/awkward-1.0/issues/176) and [#177](https://github.com/scikit-hep/awkward-1.0/issues/177)) immutable objects. Only the high-level `ak.Array` changes its state in-place in response to user choices (such as `__setitem__`, which replaces its `layout` using the pure function `ak.with_field`). This is not a performance liability but usually a benefit because it means that we can freely share data among array objects without worrying about long-distance modifications.
Arrays are immutable objects. Only the high-level `ak.Array` changes its state in-place in response to user choices (such as `__setitem__`, which replaces its `layout` using the pure function `ak.with_field`). This is not a performance liability but usually a benefit because it means that we can freely share data among array objects without worrying about long-distance modifications.

Users can break this model by wrapping NumPy arrays as Awkward Arrays and changing the original NumPy arrays in-place, but they are encouraged not to.

Expand Down
2 changes: 1 addition & 1 deletion README-pypi.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[![PyPI version](https://badge.fury.io/py/awkward.svg)](https://pypi.org/project/awkward)
[![Conda-Forge](https://img.shields.io/conda/vn/conda-forge/awkward)](https://github.com/conda-forge/awkward-feedstock)
[![Python 3.6‒3.10](https://img.shields.io/badge/python-3.6%E2%80%923.10-blue)](https://www.python.org)
[![Python 3.7‒3.11](https://img.shields.io/badge/python-3.7%E2%80%923.11-blue)](https://www.python.org)
[![BSD-3 Clause License](https://img.shields.io/badge/license-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
[![Continuous integration tests](https://img.shields.io/azure-devops/build/jpivarski/Scikit-HEP/3/main?label=tests)](https://dev.azure.com/jpivarski/Scikit-HEP/_build)

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[![PyPI version](https://badge.fury.io/py/awkward.svg)](https://pypi.org/project/awkward)
[![Conda-Forge](https://img.shields.io/conda/vn/conda-forge/awkward)](https://github.com/conda-forge/awkward-feedstock)
[![Python 3.6‒3.10](https://img.shields.io/badge/python-3.6%E2%80%923.10-blue)](https://www.python.org)
[![Python 3.7‒3.11](https://img.shields.io/badge/python-3.7%E2%80%923.11-blue)](https://www.python.org)
[![BSD-3 Clause License](https://img.shields.io/badge/license-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
[![Build Test](https://github.com/scikit-hep/awkward/actions/workflows/build-test.yml/badge.svg?branch=main)](https://github.com/scikit-hep/awkward/actions/workflows/build-test.yml)

Expand Down
4 changes: 2 additions & 2 deletions docs-sphinx/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@
:alt: Conda-Forge
:target: https://github.com/conda-forge/awkward-feedstock

.. image:: https://img.shields.io/badge/python-3.6%E2%80%923.10-blue
:alt: Python 3.6‒3.10
.. image:: https://img.shields.io/badge/python-3.7%E2%80%923.10-blue
:alt: Python 3.7‒3.10
:target: https://www.python.org

.. image:: https://img.shields.io/badge/license-BSD%203--Clause-blue.svg
Expand Down
2 changes: 1 addition & 1 deletion docs-src/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ kernelspec:

[![PyPI version](https://badge.fury.io/py/awkward.svg)](https://pypi.org/project/awkward)
[![Conda-Forge](https://img.shields.io/conda/vn/conda-forge/awkward)](https://github.com/conda-forge/awkward-feedstock)
[![Python 3.6‒3.10](https://img.shields.io/badge/python-3.6%E2%80%923.10-blue)](https://www.python.org)
[![Python 3.7‒3.10](https://img.shields.io/badge/python-3.7%E2%80%923.10-blue)](https://www.python.org)
[![BSD-3 Clause License](https://img.shields.io/badge/license-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
[![Continuous integration tests](https://img.shields.io/azure-devops/build/jpivarski/Scikit-HEP/3/main?label=tests)](https://dev.azure.com/jpivarski/Scikit-HEP/_build)

Expand Down
4 changes: 2 additions & 2 deletions docs-src/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ matplotlib
uproot
uproot3
jupyter-book
jax>=0.2.7;python_version>="3.6" and sys_platform != "win32"
jaxlib>=0.1.57;python_version>="3.6" and sys_platform != "win32"
jax>=0.2.7;sys_platform != "win32"
jaxlib>=0.1.57;sys_platform != "win32"
2 changes: 1 addition & 1 deletion noxfile.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import nox

ALL_PYTHONS = ["3.6", "3.7", "3.8", "3.9", "3.10", "3.11"]
ALL_PYTHONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]

nox.options.sessions = ["lint", "tests"]

Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ PIP_ONLY_BINARY = "cmake,numpy"

[[tool.cibuildwheel.overrides]]
select = "cp3?-*"
manylinux-x86_64-image = "manylinux2010"
manylinux-x86_64-image = "manylinux2014"


[tool.pytest.ini_options]
Expand All @@ -79,7 +79,7 @@ filterwarnings = [
log_cli_level = "info"

[tool.pylint]
master.py-version = "3.6"
master.py-version = "3.7"
master.jobs = "0"
master.ignore-paths = [
"src/awkward/_typeparser/generated_parser.py"
Expand Down
8 changes: 4 additions & 4 deletions requirements-dev.txt
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
autograd
flake8
fsspec;python_version > "3.6" and sys_platform != "win32"
jax>=0.2.7;sys_platform != "win32" and python_version > "3.6" and python_version < "3.11"
jaxlib>=0.1.57,!=0.1.68;sys_platform != "win32" and python_version > "3.6" and python_version < "3.11"
fsspec;sys_platform != "win32"
jax>=0.2.7;sys_platform != "win32" and python_version < "3.11"
jaxlib>=0.1.57,!=0.1.68;sys_platform != "win32" and python_version < "3.11"
numba>=0.50.0;python_version < "3.11"
numexpr;python_version < "3.11"
pandas>=0.24.0
pyarrow>=7.0.0;python_version > "3.6" and sys_platform != "win32" and python_version < "3.11"
pyarrow>=7.0.0;sys_platform != "win32" and python_version < "3.11"
PyYAML
3 changes: 1 addition & 2 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ classifiers =
Programming Language :: Python
Programming Language :: Python :: 3
Programming Language :: Python :: 3 :: Only
Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Expand All @@ -46,7 +45,7 @@ project_urls =

[options]
packages = find:
python_requires = >=3.6
python_requires = >=3.7
include_package_data = True
package_dir =
=src
Expand Down
8 changes: 0 additions & 8 deletions src/awkward/_v2/_connect/jax/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE

import awkward as ak
import sys

try:
import jax
Expand Down Expand Up @@ -55,13 +54,6 @@ def import_jax(name="Awkward Arrays with JAX"):
if jax is None:
raise ak._v2._util.error(ModuleNotFoundError(error_message.format(name)))

if not (sys.version_info.major == 3 and sys.version_info.minor >= 7):
raise ak._v2._util.error(
AssertionError(
"Using Awkward Arrays with JAX requires Python version >= 3.7"
)
)

global pytrees_registered

if not pytrees_registered:
Expand Down