[BEAM-12000] Add Python 3.9 support. #16008

tvalentyn · 2021-11-17T21:04:32Z

Add Python 3.9 infrastructure and test suites for Apache Beam.

tvalentyn · 2021-11-17T21:05:18Z

run seed job

tvalentyn · 2021-11-17T21:15:16Z

Run Seed Job

codecov · 2021-11-17T21:28:49Z

Codecov Report

Merging #16008 (2401a11) into master (f3412a4) will increase coverage by 36.83%.
The diff coverage is 75.00%.

@@             Coverage Diff             @@
##           master   #16008       +/-   ##
===========================================
+ Coverage   46.79%   83.62%   +36.83%     
===========================================
  Files         203      452      +249     
  Lines       20044    62252    +42208     
===========================================
+ Hits         9379    52060    +42681     
- Misses       9665    10192      +527     
+ Partials     1000        0     -1000

Impacted Files	Coverage Δ
...apache_beam/typehints/native_type_compatibility.py	`86.57% <0.00%> (ø)`
sdks/python/apache_beam/__init__.py	`85.00% <100.00%> (ø)`
sdks/python/apache_beam/runners/common.py	`89.98% <100.00%> (ø)`
...apache_beam/runners/dataflow/internal/apiclient.py	`77.39% <100.00%> (ø)`
sdks/go/pkg/beam/core/util/protox/any.go
sdks/go/pkg/beam/io/filesystem/local/local.go
sdks/go/pkg/beam/core/funcx/output.go
sdks/go/pkg/beam/core/metrics/sampler.go
...o/pkg/beam/runners/dataflow/dataflowlib/metrics.go
sdks/go/pkg/beam/transforms/stats/min_switch.go
... and 649 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f3412a4...2401a11. Read the comment docs.

tvalentyn · 2021-11-17T22:21:12Z

Run Seed Job

tvalentyn · 2021-11-17T22:33:09Z

Run Python 3.9 PostCommit

tvalentyn · 2021-11-17T22:33:40Z

Run Dataflow Python ValidatesRunner

tvalentyn · 2021-11-17T23:34:07Z

Run Python Dataflow ValidatesRunner

aaltay · 2021-12-03T01:04:28Z

What is the next step on this PR?

tvalentyn · 2021-12-03T19:17:23Z

Next step is to remove changes related to Dataflow Py 3.9 support since that runner needs additional work, and check in bits related to Apache Beam 3.9 support.

robertwb · 2022-02-04T22:58:00Z

Is this still being worked on? Are there parts of this CL that we can check in sooner rather than later?

tvalentyn · 2022-02-04T23:00:46Z

yes, it's being worked on as we speak.

tvalentyn · 2022-02-07T17:22:58Z

Thanks for taking a look at BEAM-12914, @robertwb . I will exclude the 4 failing test temporarily in this PR and it should be ready to go.

tvalentyn · 2022-02-07T23:19:43Z

R: @AnandInguva

tvalentyn · 2022-02-07T23:20:41Z

Cython currently uninstallable on Py3.9 on Jenkins VMs after Ubuntu upgrade, looking into that.

AnandInguva · 2022-02-08T20:17:08Z

build.gradle.kts

 }

 tasks.register("pythonLintPreCommit") {
+  // TODO(BEAM-8890): Find a better way to specify lint and formatter tasks without hardcoding py version.


Just to verify, for this Jira, I see an issue related to updating google-cloud-java. I don't see anything on lint and formatter.

I think it should be BEAM-9980

AnandInguva · 2022-02-08T20:34:38Z

build.gradle.kts

 tasks.register("portablePythonPreCommit") {
  dependsOn(":sdks:python:test-suites:portable:py36:preCommitPy36")
-  dependsOn(":sdks:python:test-suites:portable:py37:preCommitPy37")
+  dependsOn(":sdks:python:test-suites:portable:py39:preCommitPy39")


why do we have only py36, py39?

For some test suites, the main value may be in testing the highest supported version and the lowest supported version only to save resources and reduce execution time. But, existing suite definitions not always follow this rule and sometimes test random versions. Some suites test all versions.

tvalentyn · 2022-02-08T20:47:41Z

Fixed one more issue (https://issues.apache.org/jira/browse/BEAM-13845 has details)

build.gradle.kts

AnandInguva · 2022-02-08T20:39:40Z

build.gradle.kts

  dependsOn(":sdks:python:test-suites:tox:py38:archiveFilesToLint")
  dependsOn(":sdks:python:test-suites:tox:py38:unpackFilesToLint")
  dependsOn(":sdks:python:test-suites:tox:py38:whitespacelint")
 }

 tasks.register("typescriptPreCommit") {
+  // TODO(BEAM-8890): Find a better way to specify the tasks without hardcoding py version.


Replace BEAM-8890 with BEAM-9980

AnandInguva · 2022-02-08T20:56:04Z

.github/workflows/python_tests.yml

        exclude:
          # TODO remove exclusion after issue with protobuf is solved
          # https://github.com/protocolbuffers/protobuf/issues/7765
          - os: windows-latest
-            python: 3.8
+            python: 3.9


According the PR, this is solved for 3.9[Wheels are now present for Windows] IIUC. Can you check the PR and if it solved, can we remove the exclude for Python 3.9?

Thanks for catching!

.github/workflows/python_tests.yml

tvalentyn · 2022-02-08T23:05:56Z

Thanks, PTAL, @AnandInguva

AnandInguva · 2022-02-08T23:19:13Z

LGTM!

tvalentyn · 2022-02-09T16:07:17Z

remaining test failures are unrelated to this PR. I will squash fixups and merge.

…ues.

tvalentyn · 2022-02-09T20:24:16Z

Run Seed Job

tvalentyn · 2022-02-09T20:44:22Z

Run Python 3.9 PostCommit

tvalentyn · 2022-02-09T20:51:56Z

Run Python Dataflow V2 ValidatesRunner

tvalentyn · 2022-02-09T20:52:32Z

Run Python Dataflow ValidatesRunner

tvalentyn · 2022-02-09T20:54:40Z

Run Python Dataflow ValidatesContainer

tvalentyn · 2022-02-09T20:57:08Z

Run Python Examples_Direct

tvalentyn · 2022-02-09T21:39:10Z

Run Python 3.9 PostCommit

tvalentyn · 2022-02-09T23:47:04Z

Run Python 3.9 PostCommit

tvalentyn · 2022-02-09T23:53:55Z

Run Seed Job

tvalentyn · 2022-02-10T00:06:33Z

Run Python 3.9 PostCommit

tvalentyn · 2022-02-10T00:20:32Z

Merging based on the result of local execution + prior test results.

tvalentyn force-pushed the py39_containers branch from 42ae2fd to f02959e Compare February 7, 2022 17:21

github-actions bot added build docker infra python website labels Feb 7, 2022

tvalentyn force-pushed the py39_containers branch 2 times, most recently from 92b3cf4 to 5a89115 Compare February 7, 2022 23:17

AnandInguva reviewed Feb 8, 2022

View reviewed changes

AnandInguva approved these changes Feb 8, 2022

View reviewed changes

tvalentyn force-pushed the py39_containers branch from 772399e to 9201323 Compare February 9, 2022 19:45

tvalentyn added 10 commits February 9, 2022 12:18

Add a container for Python 3.9.

f9b7ab2

Allow job submission with Python 3.9 on Dataflow runner

af19a1f

Add Python 3.9 test suites. Keep Dataflow V1 suites unchanged for now.

4c400d2

Add py3.9 Github actions suites.

31deb25

Py39 Doc updates.

1ae9da0

[BEAM-9980] Simplify run_validates_container.sh to avoid branching.

c2d9255

Update Cython to a new version that has py39 wheels.

8c0e265

[BEAM-13845] Fix comparison with potentially incomparable default val…

fd0fcae

…ues.

[BEAM-12920] Assume that bare generators types define simple generators.

e65ff00

Mark Python 3.9 as supported version.

4b2e58f

tvalentyn force-pushed the py39_containers branch from 9201323 to 4b2e58f Compare February 9, 2022 20:24

This was referenced Feb 9, 2022

[BEAM-4032]Support staging binary distributions of dependency packages #16633

Merged

[BEAM-13860] Make DoFn.infer_output_type return element type #16788

Merged

[BEAM-13868] Remove gsutil dep from hdfs IT test.

2401a11

tvalentyn merged commit b07840e into apache:master Feb 10, 2022

tvalentyn deleted the py39_containers branch February 10, 2022 00:20

[BEAM-12000] Add Python 3.9 support. #16008

[BEAM-12000] Add Python 3.9 support. #16008

Conversation

tvalentyn commented Nov 17, 2021 • edited Loading

tvalentyn commented Nov 17, 2021

tvalentyn commented Nov 17, 2021

codecov bot commented Nov 17, 2021 • edited Loading

Codecov Report

tvalentyn commented Nov 17, 2021

tvalentyn commented Nov 17, 2021

tvalentyn commented Nov 17, 2021

tvalentyn commented Nov 17, 2021

aaltay commented Dec 3, 2021

tvalentyn commented Dec 3, 2021

robertwb commented Feb 4, 2022

tvalentyn commented Feb 4, 2022

tvalentyn commented Feb 7, 2022 • edited Loading

tvalentyn commented Feb 7, 2022

tvalentyn commented Feb 7, 2022

AnandInguva Feb 8, 2022 • edited Loading

Choose a reason for hiding this comment

AnandInguva Feb 8, 2022

Choose a reason for hiding this comment

tvalentyn Feb 8, 2022

Choose a reason for hiding this comment

AnandInguva Feb 8, 2022

Choose a reason for hiding this comment

tvalentyn Feb 8, 2022

Choose a reason for hiding this comment

tvalentyn commented Feb 8, 2022

AnandInguva Feb 8, 2022

Choose a reason for hiding this comment

AnandInguva Feb 8, 2022

Choose a reason for hiding this comment

tvalentyn Feb 8, 2022

Choose a reason for hiding this comment

tvalentyn commented Feb 8, 2022

AnandInguva commented Feb 8, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 9, 2022

tvalentyn commented Feb 10, 2022

tvalentyn commented Feb 10, 2022

tvalentyn commented Nov 17, 2021 •

edited

Loading

codecov bot commented Nov 17, 2021 •

edited

Loading

tvalentyn commented Feb 7, 2022 •

edited

Loading

AnandInguva Feb 8, 2022 •

edited

Loading