Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for Python 3.11 #23848

Closed
potiuk opened this issue Oct 26, 2022 · 9 comments · Fixed by #26121
Closed

Support for Python 3.11 #23848

potiuk opened this issue Oct 26, 2022 · 9 comments · Fixed by #26121
Assignees
@AnandInguva
Labels
core done & done Issue has been reviewed after it was closed for verification, followups, etc. new feature P2 python

Comments

@potiuk
Copy link
Member

potiuk commented Oct 26, 2022 •
edited
Loading

What would you like to happen?

I know it is eaarly (Python 3.11 has just been released yesterday) but we are hoping in Apache Airflow to a much faster cycle of adding new Python releases - especially that Pyhon 3.11 introduces huge performance improvements (25% is the average number claimed) due to a very focused effort to increase single-threaded Python performance (Specialized interpreter being the core of it but also many other improvements) without actually changing any of the Python code.

Apache Beam was traditionally one of the latest dependencies of Airflow to support newer Python version so this issue is here to make you aware that we are eaager (also to help if needed) to make it faster, possibly by talking to dependencies of Beam as well (who are oftten also direct dependencies of Airflow) and help them as well. I perfectly understand we need to bubble up support so that your dependencies support them first.

Nice summary of Py3.11 support is here: https://pyreadiness.org/3.11/ - it's not very green obviously, but I hope it gets greener soon.

I'd appreciate if someone in the Beam team attempted to migrate and have a PR running (and failing possibly until all prerequisite are met so that we can also track the progress and possibly help each other to solve any problems.

I just opened such PR in Apache Airflow yesterday and plan to keep it open until it gets green :)

apache/airflow#27264

I think it would be fantastic if we could as the open source community migrate to the new Python much faster.

Looking forward to cooperation on that one :)

Issue Priority

Priority: 3

Issue Component

Component: sdk-py-core
@potiuk potiuk mentioned this issue Oct 26, 2022
Merged
11 tasks
potiuk added a commit to apache/airflow that referenced this issue Oct 26, 2022
@potiuk
Attempt to add Python 3.11 support
cf340d7 
Python 3.11 has been released as scheduled on October 25, 2022 and
this is the first attempt to see how far Airflow (mostly dependencies)
are from being ready to officially support 3.11.

So far we had to exclude the following dependencies:

- [ ] Pyarrow dependency: apache/arrow#14499
- [ ] Google Provider: #27292
  and googleapis/python-bigquery#1386
- [ ] Databricks Provider:
  databricks/databricks-sql-python#59
- [ ] Papermill Provider: nteract/papermill#700
- [ ] Azure Provider: Azure/azure-uamqp-python#334
  and Azure/azure-sdk-for-python#27066
- [ ] Apache Beam Provider: apache/beam#23848
- [ ] Snowflake Provider:
  snowflakedb/snowflake-connector-python#1294
- [ ] JDBC Provider: jpype-project/jpype#1087
- [ ] Hive Provider: cloudera/python-sasl#30

We might decide to release Airflow in 3.11 with those providers
disabled in case they are lagging behind eventually, but for the
moment we want to work with all the projects in concert to be
able to release all providers (Google Provider requires quite
a lot of work and likely Google Team stepping up and community helping
with migration to latest Goofle cloud libraries)
potiuk added a commit to apache/airflow that referenced this issue Oct 27, 2022
@potiuk
Attempt to add Python 3.11 support
bdd765d 
Python 3.11 has been released as scheduled on October 25, 2022 and
this is the first attempt to see how far Airflow (mostly dependencies)
are from being ready to officially support 3.11.

So far we had to exclude the following dependencies:

- [ ] Pyarrow dependency: apache/arrow#14499
- [ ] Google Provider: #27292
  and googleapis/python-bigquery#1386
- [ ] Databricks Provider:
  databricks/databricks-sql-python#59
- [ ] Papermill Provider: nteract/papermill#700
- [ ] Azure Provider: Azure/azure-uamqp-python#334
  and Azure/azure-sdk-for-python#27066
- [ ] Apache Beam Provider: apache/beam#23848
- [ ] Snowflake Provider:
  snowflakedb/snowflake-connector-python#1294
- [ ] JDBC Provider: jpype-project/jpype#1087
- [ ] Hive Provider: cloudera/python-sasl#30

We might decide to release Airflow in 3.11 with those providers
disabled in case they are lagging behind eventually, but for the
moment we want to work with all the projects in concert to be
able to release all providers (Google Provider requires quite
a lot of work and likely Google Team stepping up and community helping
with migration to latest Goofle cloud libraries)
potiuk added a commit to apache/airflow that referenced this issue Oct 27, 2022
@potiuk
Attempt to add Python 3.11 support
71cbff2 
Python 3.11 has been released as scheduled on October 25, 2022 and
this is the first attempt to see how far Airflow (mostly dependencies)
are from being ready to officially support 3.11.

So far we had to exclude the following dependencies:

- [ ] Pyarrow dependency: apache/arrow#14499
- [ ] Google Provider: #27292
  and googleapis/python-bigquery#1386
- [ ] Databricks Provider:
  databricks/databricks-sql-python#59
- [ ] Papermill Provider: nteract/papermill#700
- [ ] Azure Provider: Azure/azure-uamqp-python#334
  and Azure/azure-sdk-for-python#27066
- [ ] Apache Beam Provider: apache/beam#23848
- [ ] Snowflake Provider:
  snowflakedb/snowflake-connector-python#1294
- [ ] JDBC Provider: jpype-project/jpype#1087
- [ ] Hive Provider: cloudera/python-sasl#30

We might decide to release Airflow in 3.11 with those providers
disabled in case they are lagging behind eventually, but for the
moment we want to work with all the projects in concert to be
able to release all providers (Google Provider requires quite
a lot of work and likely Google Team stepping up and community helping
with migration to latest Goofle cloud libraries)
@Abacn
Copy link
Contributor

Abacn commented Oct 31, 2022

.remove-labels 'awaiting triage'

potiuk added a commit to apache/airflow that referenced this issue Oct 31, 2022
@potiuk
Attempt to add Python 3.11 support
0463899 
Python 3.11 has been released as scheduled on October 25, 2022 and
this is the first attempt to see how far Airflow (mostly dependencies)
are from being ready to officially support 3.11.

So far we had to exclude the following dependencies:

- [ ] Pyarrow dependency: apache/arrow#14499
- [ ] Google Provider: #27292
  and googleapis/python-bigquery#1386
- [ ] Databricks Provider:
  databricks/databricks-sql-python#59
- [ ] Papermill Provider: nteract/papermill#700
- [ ] Azure Provider: Azure/azure-uamqp-python#334
  and Azure/azure-sdk-for-python#27066
- [ ] Apache Beam Provider: apache/beam#23848
- [ ] Snowflake Provider:
  snowflakedb/snowflake-connector-python#1294
- [ ] JDBC Provider: jpype-project/jpype#1087
- [ ] Hive Provider: cloudera/python-sasl#30

We might decide to release Airflow in 3.11 with those providers
disabled in case they are lagging behind eventually, but for the
moment we want to work with all the projects in concert to be
able to release all providers (Google Provider requires quite
a lot of work and likely Google Team stepping up and community helping
with migration to latest Goofle cloud libraries)
@AnandInguva
Copy link
Contributor

Thanks for opening this. This is interesting. I can open a draft PR and see what happens when I get some time.

@potiuk
Copy link
Member Author

potiuk commented Nov 1, 2022

Looking forward to it :)

@AnandInguva
Copy link
Contributor

.take-issue

@potiuk
Copy link
Member Author

potiuk commented Nov 23, 2022

FYI. Pyarrow just yesterday released 3.11 compatible binary relase (10.0.1), so if you are waiitng for that - wait no more.

potiuk added a commit to apache/airflow that referenced this issue Nov 24, 2022
@potiuk
Attempt to add Python 3.11 support
af33700 
Python 3.11 has been released as scheduled on October 25, 2022 and
this is the first attempt to see how far Airflow (mostly dependencies)
are from being ready to officially support 3.11.

So far we had to exclude the following dependencies:

- [ ] Pyarrow dependency: apache/arrow#14499
- [ ] Google Provider: #27292
  and googleapis/python-bigquery#1386
- [ ] Databricks Provider:
  databricks/databricks-sql-python#59
- [ ] Papermill Provider: nteract/papermill#700
- [ ] Azure Provider: Azure/azure-uamqp-python#334
  and Azure/azure-sdk-for-python#27066
- [ ] Apache Beam Provider: apache/beam#23848
- [ ] Snowflake Provider:
  snowflakedb/snowflake-connector-python#1294
- [ ] JDBC Provider: jpype-project/jpype#1087
- [ ] Hive Provider: cloudera/python-sasl#30

We might decide to release Airflow in 3.11 with those providers
disabled in case they are lagging behind eventually, but for the
moment we want to work with all the projects in concert to be
able to release all providers (Google Provider requires quite
a lot of work and likely Google Team stepping up and community helping
with migration to latest Goofle cloud libraries)
@AnandInguva AnandInguva mentioned this issue Dec 20, 2022
Closed
15 tasks
potiuk added a commit to potiuk/airflow that referenced this issue Jan 19, 2023
@potiuk
Attempt to add Python 3.11 support
ff8aebf 
Python 3.11 has been released as scheduled on October 25, 2022 and
this is the first attempt to see how far Airflow (mostly dependencies)
are from being ready to officially support 3.11.

So far we had to exclude the following dependencies:

- [ ] Pyarrow dependency: apache/arrow#14499
- [ ] Google Provider: apache#27292
  and googleapis/python-bigquery#1386
- [ ] Databricks Provider:
  databricks/databricks-sql-python#59
- [ ] Papermill Provider: nteract/papermill#700
- [ ] Azure Provider: Azure/azure-uamqp-python#334
  and Azure/azure-sdk-for-python#27066
- [ ] Apache Beam Provider: apache/beam#23848
- [ ] Snowflake Provider:
  snowflakedb/snowflake-connector-python#1294
- [ ] JDBC Provider: jpype-project/jpype#1087
- [ ] Hive Provider: cloudera/python-sasl#30

We might decide to release Airflow in 3.11 with those providers
disabled in case they are lagging behind eventually, but for the
moment we want to work with all the projects in concert to be
able to release all providers (Google Provider requires quite
a lot of work and likely Google Team stepping up and community helping
with migration to latest Goofle cloud libraries)
@tvalentyn tvalentyn added P1 and removed P3 labels Feb 16, 2023
@AnandInguva
Copy link
Contributor

AnandInguva commented Feb 22, 2023 •
edited
Loading

Note: I will update the blockers when I discover them

Blockers

  1. Dill 0.3.6 is required for Python 3.11 - AttributeError: 'code' object has no attribute 'co_endlinetable' with python 3.11 uqfoundation/dill#514

Updated:

Add a patch for missing Coders in #26121. But this keeps dill in the version 0.3.1.1

@kennknowles
Copy link
Member

Per our bug priorities, I think this is "normal work" aka P2

@kennknowles kennknowles added P2 and removed P1 labels Mar 23, 2023
@AnandInguva AnandInguva mentioned this issue Mar 28, 2023
Merged
3 tasks
@AnandInguva AnandInguva mentioned this issue Apr 5, 2023
Merged
@AnandInguva
Copy link
Contributor

.close-issue

@github-actions github-actions bot closed this as completed Apr 5, 2023
@potiuk
Copy link
Member Author

potiuk commented Apr 5, 2023

Cool!

@damccorm damccorm added the done & done Issue has been reviewed after it was closed for verification, followups, etc. label Apr 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AnandInguva AnandInguva

Labels
core done & done Issue has been reviewed after it was closed for verification, followups, etc. new feature P2 python
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support Python 3.11 (#26121) AnandInguva/beam
6 participants
@kennknowles @potiuk @Abacn @tvalentyn @AnandInguva @damccorm