Add Python 3.11 support (#27264)

Add Python 3.11 support Python 3.11 has been released as scheduled on October 25, 2022 and finally, after all dependencies got upgraded - we can support it. --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
apache · May 22, 2023 · c5597d1 · c5597d1
1 parent d25b194
commit c5597d1
Show file tree

Hide file tree

Showing 54 changed files with 1,060 additions and 796 deletions.
diff --git a/CI.rst b/CI.rst
@@ -59,10 +59,10 @@ Container Registry used as cache
 We are using GitHub Container Registry to store the results of the ``Build Images``
 workflow which is used in the ``Tests`` workflow.
 
-Currently in main version of Airflow we run tests in 4 different versions of Python (3.7, 3.8, 3.9, 3.10)
-which means that we have to build 8 images (4 CI ones and 4 PROD ones). Yet we run around 12 jobs
-with each of the CI images. That is a lot of time to just build the environment to run. Therefore
-we are utilising ``pull_request_target`` feature of GitHub Actions.
+Currently in main version of Airflow we run tests in all versions of Python supported,
+which means that we have to build multiple images (one CI and one PROD for each Python version).
+Yet we run many jobs (>15) - for each of the CI images. That is a lot of time to just build the
+environment to run. Therefore we are utilising ``pull_request_target`` feature of GitHub Actions.
 
 This feature allows to run a separate, independent workflow, when the main workflow is run -
 this separate workflow is different than the main one, because by default it runs using ``main`` version

diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst
@@ -51,7 +51,7 @@ Required Software Packages
 Use system-level package managers like yum, apt-get for Linux, or
 Homebrew for macOS to install required software packages:
 
-* Python (One of: 3.7, 3.8, 3.9, 3.10)
+* Python (One of: 3.7, 3.8, 3.9, 3.10, 3.11)
 * MySQL 5.7+
 * libxml
 
@@ -102,7 +102,7 @@ Creating a Local virtualenv
 
 To use your IDE for Airflow development and testing, you need to configure a virtual
 environment. Ideally you should set up virtualenv for all Python versions that Airflow
-supports (3.7, 3.8, 3.9, 3.10).
+supports (3.7, 3.8, 3.9, 3.10, 3.11).
 
 To create and initialize the local virtualenv:
 
@@ -122,7 +122,7 @@ To create and initialize the local virtualenv:
 
     .. code-block:: bash
 
-      conda create -n airflow python=3.7  # or 3.8, 3.9, 3.10
+      conda create -n airflow python=3.7  # or 3.8, 3.9, 3.10, 3.11
       conda activate airflow
 
 2. Install Python PIP requirements:
@@ -150,7 +150,7 @@ for different python versions). For development on current main source:
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     pip install -e ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 
@@ -163,7 +163,7 @@ You can also install Airflow in non-editable mode:
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     pip install ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 
@@ -173,7 +173,7 @@ sources, unless you set ``INSTALL_PROVIDERS_FROM_SOURCES`` environment variable
 
    .. code-block:: bash
 
-    # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
+    # use the same version of python as you are working with, 3.7, 3.8, 3.9, 3.10 or 3.11
     INSTALL_PROVIDERS_FROM_SOURCES="true" pip install ".[devel,<OTHER EXTRAS>]" \
         --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
 

diff --git a/README.md b/README.md
@@ -86,15 +86,15 @@ Airflow is not a streaming solution, but it is often used to process real-time d
 
 Apache Airflow is tested with:
 
-|            | Main version (dev)           | Stable version (2.6.1) |
-|------------|------------------------------|------------------------|
-| Python     | 3.7, 3.8, 3.9, 3.10          | 3.7, 3.8, 3.9, 3.10    |
-| Platform   | AMD64/ARM64(\*)              | AMD64/ARM64(\*)        |
-| Kubernetes | 1.23, 1.24, 1.25, 1.26, 1.27 | 1.23, 1.24, 1.25, 1.26 |
-| PostgreSQL | 11, 12, 13, 14, 15           | 11, 12, 13, 14, 15     |
-| MySQL      | 5.7, 8                       | 5.7, 8                 |
-| SQLite     | 3.15.0+                      | 3.15.0+                |
-| MSSQL      | 2017(\*), 2019(\*)           | 2017(\*), 2019(\*)     |
+|            | Main version (dev)           | Stable version (2.6.1)    |
+|------------|------------------------------|---------------------------|
+| Python     | 3.7, 3.8, 3.9, 3.10, 3.11    | 3.7, 3.8, 3.9, 3.10, 3.11 |
+| Platform   | AMD64/ARM64(\*)              | AMD64/ARM64(\*)           |
+| Kubernetes | 1.23, 1.24, 1.25, 1.26, 1.27 | 1.23, 1.24, 1.25, 1.26    |
+| PostgreSQL | 11, 12, 13, 14, 15           | 11, 12, 13, 14, 15        |
+| MySQL      | 5.7, 8                       | 5.7, 8                    |
+| SQLite     | 3.15.0+                      | 3.15.0+                   |
+| MSSQL      | 2017(\*), 2019(\*)           | 2017(\*), 2019(\*)        |
 
 \* Experimental
 

diff --git a/airflow/__init__.py b/airflow/__init__.py
@@ -72,6 +72,7 @@
 PY38 = sys.version_info >= (3, 8)
 PY39 = sys.version_info >= (3, 9)
 PY310 = sys.version_info >= (3, 10)
+PY311 = sys.version_info >= (3, 11)
 
 # Things to lazy import in form {local_name: ('target_module', 'target_name')}
 __lazy_imports: dict[str, tuple[str, str]] = {

diff --git a/airflow/providers/amazon/aws/hooks/dms.py b/airflow/providers/amazon/aws/hooks/dms.py
@@ -207,7 +207,7 @@ def wait_for_task_status(self, replication_task_arn: str, status: DmsTaskWaiterS
             raise TypeError("Status must be an instance of DmsTaskWaiterStatus")
 
         dms_client = self.get_conn()
-        waiter = dms_client.get_waiter(f"replication_task_{status}")
+        waiter = dms_client.get_waiter(f"replication_task_{status.value}")
         waiter.wait(
             Filters=[
                 {

diff --git a/airflow/providers/apache/hive/provider.yaml b/airflow/providers/apache/hive/provider.yaml
@@ -63,6 +63,11 @@ dependencies:
   - sasl>=0.3.1; python_version>="3.9"
   - thrift>=0.9.2
 
+# Excluded because python-sasl is not yet compatible
+# with 3.11. See https://github.com/cloudera/python-sasl/issues/30
+excluded-python-versions:
+  - "3.11"
+
 integrations:
   - integration-name: Apache Hive
     external-doc-url: https://hive.apache.org/

diff --git a/airflow/providers/apache/hive/transfers/mysql_to_hive.py b/airflow/providers/apache/hive/transfers/mysql_to_hive.py
@@ -19,6 +19,7 @@
 from __future__ import annotations
 
 from collections import OrderedDict
+from contextlib import closing
 from tempfile import NamedTemporaryFile
 from typing import TYPE_CHECKING, Sequence
 
@@ -131,28 +132,25 @@ def type_map(cls, mysql_type: int) -> str:
     def execute(self, context: Context):
         hive = HiveCliHook(hive_cli_conn_id=self.hive_cli_conn_id, auth=self.hive_auth)
         mysql = MySqlHook(mysql_conn_id=self.mysql_conn_id)
-
         self.log.info("Dumping MySQL query results to local file")
-        conn = mysql.get_conn()
-        cursor = conn.cursor()
-        cursor.execute(self.sql)
         with NamedTemporaryFile("wb") as f:
-            csv_writer = csv.writer(
-                f,
-                delimiter=self.delimiter,
-                quoting=self.quoting,
-                quotechar=self.quotechar,
-                escapechar=self.escapechar,
-                encoding="utf-8",
-            )
-            field_dict = OrderedDict()
-            if cursor.description is not None:
-                for field in cursor.description:
-                    field_dict[field[0]] = self.type_map(field[1])
-            csv_writer.writerows(cursor)
+            with closing(mysql.get_conn()) as conn:
+                with closing(conn.cursor()) as cursor:
+                    cursor.execute(self.sql)
+                    csv_writer = csv.writer(
+                        f,
+                        delimiter=self.delimiter,
+                        quoting=self.quoting,
+                        quotechar=self.quotechar if self.quoting != csv.QUOTE_NONE else None,
+                        escapechar=self.escapechar,
+                        encoding="utf-8",
+                    )
+                    field_dict = OrderedDict()
+                    if cursor.description is not None:
+                        for field in cursor.description:
+                            field_dict[field[0]] = self.type_map(field[1])
+                    csv_writer.writerows(cursor)
             f.flush()
-            cursor.close()
-            conn.close()  # type: ignore[misc]
             self.log.info("Loading file into Hive")
             hive.load_file(
                 f.name,

diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
@@ -201,7 +201,7 @@ def add_triggerer_suffix(full_path, job_id=None):
         triggerer instances.
         """
         full_path = Path(full_path).as_posix()
-        full_path += f".{LogType.TRIGGER}"
+        full_path += f".{LogType.TRIGGER.value}"
         if job_id:
             full_path += f".{job_id}.log"
         return full_path

diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
@@ -670,7 +670,7 @@ the older branches, you should set the "skip" field to true.
 ## Verify production images
 
 ```shell script
-for PYTHON in 3.7 3.8 3.9 3.10
+for PYTHON in 3.7 3.8 3.9 3.10 3.11
 do
     docker pull apache/airflow:${VERSION}-python${PYTHON}
     breeze prod-image verify --image-name apache/airflow:${VERSION}-python${PYTHON}

diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py
@@ -36,7 +36,7 @@
 APACHE_AIRFLOW_GITHUB_REPOSITORY = "apache/airflow"
 
 # Checked before putting in build cache
-ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 DEFAULT_PYTHON_MAJOR_MINOR_VERSION = ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS[0]
 ALLOWED_ARCHITECTURES = [Architecture.X86_64, Architecture.ARM]
 ALLOWED_BACKENDS = ["sqlite", "mysql", "postgres", "mssql"]
@@ -174,7 +174,7 @@ def get_default_platform_machine() -> str:
 PYTHONDONTWRITEBYTECODE = True
 
 PRODUCTION_IMAGE = False
-ALL_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALL_PYTHON_MAJOR_MINOR_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 CURRENT_PYTHON_MAJOR_MINOR_VERSIONS = ALL_PYTHON_MAJOR_MINOR_VERSIONS
 CURRENT_POSTGRES_VERSIONS = ["11", "12", "13", "14", "15"]
 DEFAULT_POSTGRES_VERSION = CURRENT_POSTGRES_VERSIONS[0]

diff --git a/dev/breeze/tests/test_cache.py b/dev/breeze/tests/test_cache.py
@@ -36,8 +36,8 @@
     [
         ("backend", "mysql", (True, ["sqlite", "mysql", "postgres", "mssql"]), None),
         ("backend", "xxx", (False, ["sqlite", "mysql", "postgres", "mssql"]), None),
-        ("python_major_minor_version", "3.8", (True, ["3.7", "3.8", "3.9", "3.10"]), None),
-        ("python_major_minor_version", "3.5", (False, ["3.7", "3.8", "3.9", "3.10"]), None),
+        ("python_major_minor_version", "3.8", (True, ["3.7", "3.8", "3.9", "3.10", "3.11"]), None),
+        ("python_major_minor_version", "3.5", (False, ["3.7", "3.8", "3.9", "3.10", "3.11"]), None),
         ("missing", "value", None, AttributeError),
     ],
 )

diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py
@@ -266,10 +266,10 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
                 ("setup.py",),
                 {
                     "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "needs-helm-tests": "true",
                     "run-tests": "true",
@@ -289,10 +289,10 @@ def assert_outputs_are_printed(expected_outputs: dict[str, str], stderr: str):
                 ("generated/provider_dependencies.json",),
                 {
                     "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "needs-helm-tests": "true",
                     "run-tests": "true",
@@ -397,10 +397,10 @@ def test_expected_output_pull_request_main(
                 "main",
                 {
                     "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -424,10 +424,10 @@ def test_expected_output_pull_request_main(
                 "main",
                 {
                     "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -449,10 +449,10 @@ def test_expected_output_pull_request_main(
                 "main",
                 {
                     "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -474,10 +474,10 @@ def test_expected_output_pull_request_main(
                 "v2-3-stable",
                 {
                     "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
-                    "python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                    "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
+                    "python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                    "python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                     "image-build": "true",
                     "run-tests": "true",
                     "docs-build": "true",
@@ -787,8 +787,8 @@ def test_expected_output_pull_request_target(
             "main",
             {
                 "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "true",
                 "run-tests": "true",
@@ -807,8 +807,8 @@ def test_expected_output_pull_request_target(
             "v2-3-stable",
             {
                 "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "false",
                 "run-tests": "true",
@@ -826,8 +826,8 @@ def test_expected_output_pull_request_target(
             "main",
             {
                 "affected-providers-list-as-string": ALL_PROVIDERS_AFFECTED,
-                "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+                "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+                "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
                 "image-build": "true",
                 "needs-helm-tests": "true",
                 "run-tests": "true",
@@ -878,8 +878,8 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(github_event):
     )
     assert_outputs_are_printed(
         {
-            "all-python-versions": "['3.7', '3.8', '3.9', '3.10']",
-            "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10",
+            "all-python-versions": "['3.7', '3.8', '3.9', '3.10', '3.11']",
+            "all-python-versions-list-as-string": "3.7 3.8 3.9 3.10 3.11",
             "image-build": "true",
             "needs-helm-tests": "true",
             "run-tests": "true",

diff --git a/dev/provider_packages/prepare_provider_packages.py b/dev/provider_packages/prepare_provider_packages.py
@@ -52,7 +52,7 @@
 from rich.syntax import Syntax
 from yaml import safe_load
 
-ALL_PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+ALL_PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 
 MIN_AIRFLOW_VERSION = "2.4.0"
 # In case you have some providers that you want to have different min-airflow version for,

diff --git a/dev/retag_docker_images.py b/dev/retag_docker_images.py
@@ -31,7 +31,7 @@
 
 import rich_click as click
 
-PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10"]
+PYTHON_VERSIONS = ["3.7", "3.8", "3.9", "3.10", "3.11"]
 
 GHCR_IO_PREFIX = "ghcr.io"
 

diff --git a/docs/apache-airflow/howto/upgrading-from-1-10/index.rst b/docs/apache-airflow/howto/upgrading-from-1-10/index.rst
@@ -34,7 +34,7 @@ Airflow 1.10 was the last release series to support Python 2. Airflow 2.0.0
 requires Python 3.6+ and has been tested with Python versions 3.6, 3.7 and 3.8.
 Python 3.9 support was added from Airflow 2.1.2.
 
-Airflow 2.3.0 dropped support for Python 3.6. It's tested with Python 3.7, 3.8, 3.9 and 3.10.
+Airflow 2.3.0 dropped support for Python 3.6. It's tested with Python 3.7, 3.8, 3.9, 3.10, 3.11.
 
 If you have a specific task that still requires Python 2 then you can use the ``@task.virtualenv``, ``@task.docker`` or ``@task.kubernetes`` decorators for this.
 

diff --git a/docs/apache-airflow/installation/prerequisites.rst b/docs/apache-airflow/installation/prerequisites.rst
@@ -20,7 +20,7 @@ Prerequisites
 
 Starting with Airflow 2.3.0, Airflow is tested with:.
 
-* Python: 3.7, 3.8, 3.9, 3.10
+* Python: 3.7, 3.8, 3.9, 3.10, 3.11
 
 * Databases: