Python 3.10 support #23587

AnandInguva · 2022-10-11T22:42:37Z

Fixes: #21458
Fixes: #21671

This PR covers

Python 3.10 tests and infrastructure.
replace _unified_repr() with repr(). Fixed the tests related to _unified_repr() Fixes: Replace _unified_repr with repr #20982
Some tests were meant to be run on Dataflow runner v1 when it was the default runner for Dataflow. But Runner v2 has been added as default so we need to explicitly add --experiments=disable_runner_v2 or --dataflow_service_option=disable_runner_v2. Added this flag to dataflow runner v1 tests.

Note: Also, dataflow runner v1 is not supported yet with Python 3.10. Please use runner_v2 for the Dataflow jobs.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

update python 3.10 dependencies

apache#21971

…gs.gradle.kts Add InferencePostCommit to Py310 from Py39 Change jump offset multiplier Add GEN_START Update comment

Remove unified repr (#4) * Fix NewType and NamedTuple * Fix names in unit tests Remove commented out code (#5) Fix more failing tests (#6) * Remove commented out code * Fix other tests fixup assertion resolve merge conflicts

Fixup test

Unskip tests that were sickbayed for runner_v1

Fixup: typo

codecov · 2022-10-11T23:26:03Z

Codecov Report

Merging #23587 (c6b9cdb) into master (98eeac7) will decrease coverage by 0.08%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master   #23587      +/-   ##
==========================================
- Coverage   73.40%   73.32%   -0.09%     
==========================================
  Files         719      719              
  Lines       95940    95797     -143     
==========================================
- Hits        70424    70241     -183     
- Misses      24205    24245      +40     
  Partials     1311     1311

Flag	Coverage Δ
python	`83.03% <83.33%> (-0.10%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ython/apache_beam/runners/interactive/sql/utils.py	`76.09% <ø> (ø)`
sdks/python/apache_beam/typehints/row_type.py	`100.00% <ø> (ø)`
...apache_beam/typehints/native_type_compatibility.py	`85.52% <33.33%> (-1.06%)`	⬇️
sdks/python/apache_beam/typehints/opcodes.py	`85.35% <50.00%> (-0.26%)`	⬇️
.../python/apache_beam/typehints/trivial_inference.py	`96.15% <87.50%> (-0.27%)`	⬇️
sdks/python/apache_beam/__init__.py	`85.00% <100.00%> (ø)`
...apache_beam/runners/dataflow/internal/apiclient.py	`77.34% <100.00%> (ø)`
...s/python/apache_beam/typehints/sharded_key_type.py	`93.54% <100.00%> (ø)`
sdks/python/apache_beam/typehints/typehints.py	`93.37% <100.00%> (-0.06%)`	⬇️
.../python/apache_beam/testing/test_stream_service.py	`88.09% <0.00%> (-4.77%)`	⬇️
... and 14 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

AnandInguva · 2022-10-11T23:33:28Z

R: @tvalentyn

github-actions · 2022-10-11T23:34:41Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

tvalentyn · 2022-10-12T01:32:40Z

build.gradle.kts

+}
+
+tasks.register("python310PostCommit") {
+  dependsOn(":sdks:python:test-suites:dataflow:py310:postCommitIT")


do these run on RunnerV2 by default?

Yes. Default is runner_v2

AnandInguva · 2022-10-12T17:25:06Z

Run Python Dataflow ValidatesRunner

AnandInguva · 2022-10-12T17:40:53Z

I added dataflow runner v1 tests with explicit disable_runner_v2 option. PTAL @tvalentyn. Also, addressed the comments

AnandInguva · 2022-10-12T18:08:53Z

Run Portable_Python PreCommit

AnandInguva · 2022-10-12T19:03:52Z

Run Python Dataflow ValidatesRunner

AnandInguva · 2022-10-12T19:04:05Z

Run Python 3.10 PostCommit

AnandInguva · 2022-10-12T19:37:06Z

Run Python Dataflow ValidatesRunner

AnandInguva · 2022-10-12T21:21:25Z

Run Python Dataflow ValidatesContainer

AnandInguva · 2022-10-13T04:03:51Z

Run Python Dataflow ValidatesContainer

tvalentyn · 2022-10-14T02:17:18Z

.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy

@@ -38,6 +38,7 @@ PostcommitJobBuilder.postCommitJob('beam_PostCommit_Py_VR_Dataflow', 'Run Python
          rootBuildScriptDir(commonJobProperties.checkoutDir)
          tasks(':sdks:python:test-suites:dataflow:validatesRunnerBatchTests')
          tasks(':sdks:python:test-suites:dataflow:validatesRunnerStreamingTests')
+          switches('-PdisableRunnerV2')


good catch!

AnandInguva · 2022-10-14T16:08:08Z

Run Python 3.10 PostCommit

AnandInguva · 2022-10-14T16:08:19Z

Run Python Dataflow ValidatesContainer

AnandInguva · 2022-10-14T16:08:29Z

Run Python Dataflow ValidatesRunner

AnandInguva · 2022-10-14T16:09:36Z

Run Python Dataflow V2 ValidatesRunner

AnandInguva · 2022-10-14T19:47:32Z

Run Python 3.10 PostCommit

AnandInguva · 2022-10-14T20:12:37Z

Run Python 3.10 PostCommit

AnandInguva · 2022-10-14T20:12:45Z

Run Python 3.9 PostCommit

AnandInguva · 2022-10-17T20:26:45Z

Run Python PreCommit

tvalentyn · 2022-10-17T22:28:00Z

Run Python Dataflow ValidatesContainer

AnandInguva · 2022-10-18T00:05:23Z

Run Python Dataflow ValidatesContainer

tvalentyn · 2022-10-18T05:01:42Z

Run Python Dataflow ValidatesContainer

tvalentyn · 2022-10-18T17:57:58Z

Remaing test failures are unrealated or uncorrectly attributed to this PR.

github-actions bot added build docker infra python website labels Oct 11, 2022

AnandInguva and others added 16 commits October 11, 2022 18:45

update dependencies with updated pytest related changes

b3d2be9

add 3.10 to tox and pytest

d262575

Change float to string for python versions

4b15953

Add python 3.10 container

115b229

update python 3.10 dependencies

Add precommit and postcommit changes

547e6bb

apache#21971

Add Python 3.10 build file to the test suites

20bf42d

Update tox with python 3.10 and add python 3.10 test suites to settin…

bee034d

…gs.gradle.kts Add InferencePostCommit to Py310 from Py39 Change jump offset multiplier Add GEN_START Update comment

Fix NewType and NamedTuple (#3)

7e870b4

Remove unified repr (#4) * Fix NewType and NamedTuple * Fix names in unit tests Remove commented out code (#5) Fix more failing tests (#6) * Remove commented out code * Fix other tests fixup assertion resolve merge conflicts

Add Python 3.10 to Dataflow supported version. Sickbay runner v1 tests

3fb9c0b

Remove Python 3.10 warning

09ae110

Fixup test

Enable additional Dataflow test suites

5333ec4

Add Python 3.10 to GH actions tests

c73ef29

Add Python 3.10 to some dataflow test-suites

63dc02f

Unskip tests that were sickbayed for runner_v1

Update pandas dep for Python 3.10

37faa0a

Fixup: typo

generate dependencies

40e2dd0

Pin protobuf

ffe094f

AnandInguva force-pushed the python_310 branch from c1cb786 to ffe094f Compare October 11, 2022 22:47

fixups: lint, refactors

90ee902

update dependencies

bed577b

AnandInguva force-pushed the python_310 branch from 423e19e to bed577b Compare October 11, 2022 23:32

AnandInguva marked this pull request as ready for review October 11, 2022 23:33

tvalentyn reviewed Oct 12, 2022

View reviewed changes

Clear comments

e306d33

Add protobuf version for MacOS using Python 3.10

bbe1c6c

tvalentyn reviewed Oct 14, 2022

View reviewed changes

AnandInguva added 2 commits October 17, 2022 12:40

build wheels for python 3.10

d26bcf4

Update md files

c6b9cdb

This was referenced Oct 18, 2022

[CI] Use quotes to avoid python-version 3.10 be interpreted as 3.1 #23694

Closed

bump to python 310 in all images pangeo-data/pangeo-docker-images#334

Merged

tvalentyn merged commit 61f7bfe into apache:master Oct 18, 2022

Abacn mentioned this pull request Oct 18, 2022

Downgrade container cryptography version to avoid yanked version #23711

Merged

4 tasks

AnandInguva mentioned this pull request Oct 21, 2022

[BEAM-13590]Add support for python 3.10 #17700

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.10 support #23587

Python 3.10 support #23587

AnandInguva commented Oct 11, 2022 •

edited

Loading

codecov bot commented Oct 11, 2022 •

edited

Loading

AnandInguva commented Oct 11, 2022

github-actions bot commented Oct 11, 2022

tvalentyn Oct 12, 2022

AnandInguva Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 13, 2022

tvalentyn Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 17, 2022

tvalentyn commented Oct 17, 2022

AnandInguva commented Oct 18, 2022

tvalentyn commented Oct 18, 2022

tvalentyn commented Oct 18, 2022

Python 3.10 support #23587

Python 3.10 support #23587

Conversation

AnandInguva commented Oct 11, 2022 • edited Loading

GitHub Actions Tests Status (on master branch)

codecov bot commented Oct 11, 2022 • edited Loading

Codecov Report

AnandInguva commented Oct 11, 2022

github-actions bot commented Oct 11, 2022

tvalentyn Oct 12, 2022

Choose a reason for hiding this comment

AnandInguva Oct 12, 2022

Choose a reason for hiding this comment

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 12, 2022

AnandInguva commented Oct 13, 2022

tvalentyn Oct 14, 2022

Choose a reason for hiding this comment

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 14, 2022

AnandInguva commented Oct 17, 2022

tvalentyn commented Oct 17, 2022

AnandInguva commented Oct 18, 2022

tvalentyn commented Oct 18, 2022

tvalentyn commented Oct 18, 2022

AnandInguva commented Oct 11, 2022 •

edited

Loading

codecov bot commented Oct 11, 2022 •

edited

Loading