Attempt to add Python 3.11 support

Python 3.11 has been released as scheduled on October 25, 2022 and this is the first attempt to see how far Airflow (mostly dependencies) are from being ready to officially support 3.11. So far we had to exclude the following dependencies: - [ ] Pyarrow dependency: apache/arrow#14499 - [ ] Google Provider: apache#27292 and googleapis/python-bigquery#1386 - [ ] Databricks Provider: databricks/databricks-sql-python#59 - [ ] Papermill Provider: nteract/papermill#700 - [ ] Azure Provider: Azure/azure-uamqp-python#334 and Azure/azure-sdk-for-python#27066 - [ ] Apache Beam Provider: apache/beam#23848 - [ ] Snowflake Provider: snowflakedb/snowflake-connector-python#1294 - [ ] JDBC Provider: jpype-project/jpype#1087 - [ ] Hive Provider: cloudera/python-sasl#30 We might decide to release Airflow in 3.11 with those providers disabled in case they are lagging behind eventually, but for the moment we want to work with all the projects in concert to be able to release all providers (Google Provider requires quite a lot of work and likely Google Team stepping up and community helping with migration to latest Goofle cloud libraries)
potiuk · Jan 19, 2023 · ff8aebf · ff8aebf
1 parent 7f2b065
commit ff8aebf
Show file tree

Hide file tree

Showing 52 changed files with 922 additions and 694 deletions.
diff --git a/CI.rst b/CI.rst
@@ -59,7 +59,7 @@ Container Registry used as cache
 We are using GitHub Container Registry to store the results of the ``Build Images``
 workflow which is used in the ``Tests`` workflow.
 
-Currently in main version of Airflow we run tests in 4 different versions of Python (3.7, 3.8, 3.9, 3.10)
+Currently in main version of Airflow we run tests in those versions of Python (3.7, 3.8, 3.9, 3.10, 3.11)
 which means that we have to build 8 images (4 CI ones and 4 PROD ones). Yet we run around 12 jobs
 with each of the CI images. That is a lot of time to just build the environment to run. Therefore
 we are utilising ``pull_request_target`` feature of GitHub Actions.

diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst
@@ -51,7 +51,7 @@ Required Software Packages
 Use system-level package managers like yum, apt-get for Linux, or
 Homebrew for macOS to install required software packages:
 
-* Python (One of: 3.7, 3.8, 3.9, 3.10)
+* Python (One of: 3.7, 3.8, 3.9, 3.10, 3.11)
 * MySQL 5.7+
 * libxml
 
@@ -102,7 +102,7 @@ Creating a Local virtualenv
 
 To use your IDE for Airflow development and testing, you need to configure a virtual
 environment. Ideally you should set up virtualenv for all Python versions that Airflow
-supports (3.7, 3.8, 3.9, 3.10).
+supports (3.7, 3.8, 3.9, 3.10, 3.11).
 
 To create and initialize the local virtualenv:
 
@@ -122,7 +122,7 @@ To create and initialize the local virtualenv:
 
     .. code-block:: bash
 
-      conda create -n airflow python=3.7  # or 3.8, 3.9, 3.10
+      conda create -n airflow python=3.7  # or 3.8, 3.9, 3.10, 3.11
       conda activate airflow
 
 2. Install Python PIP requirements:
@@ -150,7 +150,7 @@ for different python versions). For development on current main source:
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     pip install -e ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 
@@ -163,7 +163,7 @@ You can also install Airflow in non-editable mode:
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     pip install ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 
@@ -173,7 +173,7 @@ sources, unless you set ``INSTALL_PROVIDERS_FROM_SOURCES`` environment variable
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     INSTALL_PROVIDERS_FROM_SOURCES="true" pip install ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 

diff --git a/README.md b/README.md
@@ -88,7 +88,7 @@ Apache Airflow is tested with:
 
 |                     | Main version (dev)           | Stable version (2.5.0)       |
 |---------------------|------------------------------|------------------------------|
-| Python              | 3.7, 3.8, 3.9, 3.10          | 3.7, 3.8, 3.9, 3.10          |
+| Python              | 3.7, 3.8, 3.9, 3.10, 3.11    | 3.7, 3.8, 3.9, 3.10, 3.11    |
 | Platform            | AMD64/ARM64(\*)              | AMD64/ARM64(\*)              |
 | Kubernetes          | 1.23, 1.24, 1.25, 1.26       | 1.21, 1.22, 1.23, 1.24, 1.25 |
 | PostgreSQL          | 11, 12, 13, 14, 15           | 11, 12, 13, 14, 15           |

diff --git a/airflow/providers/apache/beam/provider.yaml b/airflow/providers/apache/beam/provider.yaml
@@ -40,6 +40,11 @@ dependencies:
   - apache-airflow>=2.3.0
   - apache-beam>=2.33.0
 
+# Excluded because apache-beam and a number of dependent libraries are not
+# compatible with 3.11 yet. See https://github.com/apache/beam/issues/23848
+excluded-python-versions:
+  - "3.11"
+
 integrations:
   - integration-name: Apache Beam
     external-doc-url: https://beam.apache.org/

diff --git a/airflow/providers/apache/hive/provider.yaml b/airflow/providers/apache/hive/provider.yaml
@@ -58,6 +58,11 @@ dependencies:
   - sasl>=0.3.1; python_version>="3.9"
   - thrift>=0.9.2
 
+# Excluded because SASL is not yet compatible
+# with 3.11. See https://github.com/cloudera/python-sasl/issues/30
+excluded-python-versions:
+  - "3.11"
+
 integrations:
   - integration-name: Apache Hive
     external-doc-url: https://hive.apache.org/

diff --git a/airflow/providers/databricks/provider.yaml b/airflow/providers/databricks/provider.yaml
@@ -48,6 +48,12 @@ dependencies:
   - databricks-sql-connector>=2.0.0, <3.0.0
   - aiohttp>=3.6.3, <4
 
+# Excluded because databricks-sql-connector and a number of dependent libraries are not compatible
+# with 3.11 yet See https://pyreadiness.org/3.11/. (Pyarrow is one of the important ones)
+# See https://github.com/databricks/databricks-sql-python/issues/59 for issue that tracks it
+excluded-python-versions:
+  - "3.11"
+
 integrations:
   - integration-name: Databricks
     external-doc-url: https://databricks.com/

diff --git a/airflow/providers/google/provider.yaml b/airflow/providers/google/provider.yaml
@@ -123,6 +123,12 @@ dependencies:
   # Google bigtable client require protobuf <= 3.20.0. We can remove the limitation
   # when this limitation is removed
   - protobuf<=3.20.0
+# Excluded because bigquery and a number of dependent libraries are not compatible with 3.11 yet
+# See https://pyreadiness.org/3.11/. Also some of Google libraries are in pretty ancient
+# version (pre 2.0.0) and they MUST be migrated in order to add 3.11 support
+# This is tracked in https://github.com/apache/airflow/issues/27292
+excluded-python-versions:
+  - "3.11"
 
 integrations:
   - integration-name: Google Analytics360

diff --git a/airflow/providers/papermill/provider.yaml b/airflow/providers/papermill/provider.yaml
@@ -40,6 +40,13 @@ dependencies:
   - papermill[all]>=1.2.1
   - scrapbook[all]
 
+# Excluded because papermill and a number of dependent libraries are not compatible
+# with 3.11 yet See https://pyreadiness.org/3.11/. (Pyarrow is one of the important ones)
+# See for issue that tracks it https://github.com/nteract/papermill/issues/700
+excluded-python-versions:
+  - "3.11"
+
+
 integrations:
   - integration-name: Papermill
     external-doc-url: https://github.com/nteract/papermill

diff --git a/airflow/providers/snowflake/provider.yaml b/airflow/providers/snowflake/provider.yaml
@@ -53,6 +53,13 @@ dependencies:
   - snowflake-connector-python>=2.4.1
   - snowflake-sqlalchemy>=1.1.0
 
+# Excluded because snowflake connector python and a number of dependent libraries are not
+# compatible with 3.11 yet. See https://pyreadiness.org/3.11/. One of the important dependencies
+# is Pyarrow: https://github.com/apache/arrow/pull/14499 but the
+# https://github.com/snowflakedb/snowflake-connector-python/issues/1289 is the issue
+excluded-python-versions:
+  - "3.11"
+
 integrations:
   - integration-name: Snowflake
     external-doc-url: https://snowflake.com/

diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
@@ -1036,7 +1036,7 @@ the older branches, you should set the "skip" field to true.
 ## Verify production images
 
 ```shell script
-for PYTHON in 3.7 3.8 3.9 3.10
+for PYTHON in 3.7 3.8 3.9 3.10 3.11
 do
     docker pull apache/airflow:${VERSION}-python${PYTHON}
     breeze prod-image verify --image-name apache/airflow:${VERSION}-python${PYTHON}

diff --git a/dev/breeze/SELECTIVE_CHECKS.md b/dev/breeze/SELECTIVE_CHECKS.md
@@ -108,8 +108,8 @@ The selective check outputs available are described below:
 
 | Output                             | Meaning of the output                                                                                  | Example value                                                 |
 |------------------------------------|--------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|
-| all-python-versions                | List of all python versions there are available in the form of JSON array                              | ['3.7', '3.8', '3.9', '3.10']                                 |
-| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string                  | 3.7 3.8 3.9 3.10                                              |
+| all-python-versions                | List of all python versions there are available in the form of JSON array                              | ['3.7', '3.8', '3.9', '3.10', '3.11']                         |
+| all-python-versions-list-as-string | List of all python versions there are available in the form of space separated string                  | 3.7 3.8 3.9 3.10 3.11                                         |
 | basic-checks-only                  | Whether to run all static checks ("false") or only basic set of static checks ("true")                 | false                                                         |
 | cache-directive                    | Which cache should be be used for images ("registry", "local" , "disabled")                            | registry                                                      |
 | debug-resources                    | Whether resources usage should be printed during parallel job execution ("true"/ "false")              | false                                                         |

diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -36,7 +36,7 @@
 APACHE_AIRFLOW_GITHUB_REPOSITORY = "apache/airflow"
 
 # Checked before putting in build cache
-ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 DEFAULT_PYTHON_MAJOR_MINOR_VERSION = ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS[0]
 ALLOWED_ARCHITECTURES = [Architecture.X86_64, Architecture.ARM]
 ALLOWED_BACKENDS = ["sqlite", "mysql", "postgres", "mssql"]
@@ -148,7 +148,7 @@ def get_default_platform_machine() -> str:
 PYTHONDONTWRITEBYTECODE = True
 
 PRODUCTION_IMAGE = False
-ALL_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALL_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 CURRENT_PYTHON_MAJOR_MINOR_VERSIONS = ALL_PYTHON_MAJOR_MINOR_VERSIONS
 CURRENT_POSTGRES_VERSIONS = ["11", "12", "13", "14", "15"]
 DEFAULT_POSTGRES_VERSION = CURRENT_POSTGRES_VERSIONS[0]

diff --git a/dev/breeze/tests/test_cache.py b/dev/breeze/tests/test_cache.py
@@ -36,8 +36,8 @@
     [
         ("backend", "mysql", (True, ["sqlite", "mysql", "postgres", "mssql"]), None),
         ("backend", "xxx", (False, ["sqlite", "mysql", "postgres", "mssql"]), None),
-        ("python_major_minor_version", "3.8", (True, ["3.7", "3.8", "3.9", "3.10"]), None),
-        ("python_major_minor_version", "3.5", (False, ["3.7", "3.8", "3.9", "3.10"]), None),
+        ("python_major_minor_version", "3.8", (True, ["3.7", "3.8", "3.9", "3.10", "3.11"]), None),
+        ("python_major_minor_version", "3.5", (False, ["3.7", "3.8", "3.9", "3.10", "3.11"]), None),
         ("missing", "value", None, AttributeError),
     ],
 )

diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
@@ -223,10 +223,10 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
             pytest.param(
                 ("setup.py",),
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "needs-helm-tests": "true",
                     "run-tests": "true",
@@ -242,10 +242,10 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
             pytest.param(
                 ("generated/provider_dependencies.json",),
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "needs-helm-tests": "true",
                     "run-tests": "true",
@@ -281,10 +281,10 @@ def test_expected_output_pull_request_main(
                 ("full tests needed",),
                 "main",
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -305,10 +305,10 @@ def test_expected_output_pull_request_main(
                 ),
                 "main",
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -327,10 +327,10 @@ def test_expected_output_pull_request_main(
                 ("full tests needed",),
                 "main",
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -349,10 +349,10 @@ def test_expected_output_pull_request_main(
                 ("full tests needed",),
                 "v2-3-stable",
                 {
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -570,8 +570,8 @@ def test_expected_output_pull_request_target(
             (),
             "main",
             {
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "true",
                 "run-tests": "true",
@@ -586,8 +586,8 @@ def test_expected_output_pull_request_target(
             (),
             "v2-3-stable",
             {
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "false",
                 "run-tests": "true",
@@ -603,8 +603,8 @@ def test_expected_output_pull_request_target(
             (),
             "main",
             {
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "true",
                 "run-tests": "true",
@@ -652,8 +652,8 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(github_event):
     )
     assert_outputs_are_printed(
         {
-            "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-            "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+            "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+            "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
             "image-build": "true",
             "needs-helm-tests": "true",
             "run-tests": "true",

diff --git a/dev/provider_packages/prepare_provider_packages.py b/dev/provider_packages/prepare_provider_packages.py
@@ -51,7 +51,7 @@
 from rich.syntax import Syntax
 from yaml import safe_load
 
-ALL_PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALL_PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 
 INITIAL_CHANGELOG_CONTENT = """
 

diff --git a/dev/retag_docker_images.py b/dev/retag_docker_images.py
@@ -31,7 +31,7 @@
 
 import rich_click as click
 
-PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 
 GHCR_IO_PREFIX = "ghcr.io"
 

diff --git a/docs/apache-airflow/howto/upgrading-from-1-10/index.rst b/docs/apache-airflow/howto/upgrading-from-1-10/index.rst
@@ -34,7 +34,7 @@ Airflow 1.10 was the last release series to support Python 2. Airflow 2.0.0
 requires Python 3.6+ and has been tested with Python versions 3.6, 3.7 and 3.8.
 Python 3.9 support was added from Airflow 2.1.2.
 
-Airflow 2.3.0 dropped support for Python 3.6. It's tested with Python 3.7, 3.8, 3.9 and 3.10.
+Airflow 2.3.0 dropped support for Python 3.6. It's tested with Python 3.7, 3.8, 3.9, 3.10, 3.11.
 
 If you have a specific task that still requires Python 2 then you can use the ``@task.virtualenv``, ``@task.docker`` or ``@task.kubernetes`` decorators for this.
 

diff --git a/docs/apache-airflow/installation/prerequisites.rst b/docs/apache-airflow/installation/prerequisites.rst
@@ -20,7 +20,7 @@ Prerequisites
 
 Starting with Airflow 2.3.0, Airflow is tested with:.
 
-* Python: 3.7, 3.8, 3.9, 3.10
+* Python: 3.7, 3.8, 3.9, 3.10, 3.11
 
 * Databases: