Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: fix invalid escape sequences #11147

Merged
merged 1 commit into from
Sep 20, 2024

Conversation

TheKevJames
Copy link
Contributor

Description of your changes:
This PR fixes SyntaxError issues in Python3.12 and deprecations in previous versions.

In python3.12, using escape sequences outside of raw strings has been moved from a deprecation warning into a syntax error. As such, I've added a linter to track down cases where we still have this issue and fixed all said reported issues.

The original list was as follows:
backend/src/apiserver/visualization/types/tfdv.py:78:122: W605 invalid escape sequence '\/'
components/aws/athena/query/src/query.py:66:42: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/container/v1/aiplatform/remote_runner.py:91:44: W605 invalid escape sequence '\w'
components/google-cloud/google_cloud_pipeline_components/container/v1/aiplatform/remote_runner.py:91:46: W605 invalid escape sequence '\-'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:431: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:433: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:440: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:468: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:470: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:483: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:516: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:518: W605 invalid escape sequence '\/'
components/google-cloud/google_cloud_pipeline_components/v1/hyperparameter_tuning_job/component.py:53:524: W605 invalid escape sequence '\/'
proxy/get_proxy_url.py:43:46: W605 invalid escape sequence '\d'
proxy/get_proxy_url.py:58:47: W605 invalid escape sequence '\d'
sdk/python/kfp/compiler/compiler_test.py:4448:21: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4448:70: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4448:81: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4448:87: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4911:106: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4911:132: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4911:178: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4911:182: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4964:106: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4964:132: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4964:178: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4964:191: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4986:70: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4986:137: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4986:198: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4986:301: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:4986:313: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:5009:106: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:5009:132: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:5009:178: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:5009:182: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:5146:22: W605 invalid escape sequence '\.'
sdk/python/kfp/compiler/compiler_test.py:5146:67: W605 invalid escape sequence '\.'
sdk/python/kfp/deprecated/compiler/v2_compatible_compiler_test.py:91:40: W605 invalid escape sequence '\d'
sdk/python/kfp/deprecated/compiler/v2_compatible_compiler_test.py:91:46: W605 invalid escape sequence '\d'
sdk/python/kfp/deprecated/compiler/v2_compatible_compiler_test.py:91:52: W605 invalid escape sequence '\d'
sdk/python/kfp/deprecated/compiler/v2_compatible_compiler_test.py:157:49: W605 invalid escape sequence '\('
sdk/python/kfp/deprecated/compiler/v2_compatible_compiler_test.py:158:54: W605 invalid escape sequence '\)'
sdk/python/kfp/deprecated/components/type_annotation_utils.py:61:30: W605 invalid escape sequence '\.'
sdk/python/kfp/deprecated/components/type_annotation_utils.py:61:43: W605 invalid escape sequence '\w'
sdk/python/kfp/deprecated/components/type_annotation_utils.py:61:50: W605 invalid escape sequence '\['
sdk/python/kfp/deprecated/components/type_annotation_utils.py:61:54: W605 invalid escape sequence '\]'
sdk/python/kfp/deprecated/components_tests/test_python_op.py:1227:83: W605 invalid escape sequence '\['
sdk/python/kfp/deprecated/components_tests/test_python_op.py:1227:88: W605 invalid escape sequence '\]'
sdk/python/kfp/deprecated/dsl/_ops_group.py:71:27: W605 invalid escape sequence '\d'
sdk/python/kfp/dsl/executor.py:403:30: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/executor.py:403:43: W605 invalid escape sequence '\w'
sdk/python/kfp/dsl/executor.py:403:50: W605 invalid escape sequence '\['
sdk/python/kfp/dsl/executor.py:403:54: W605 invalid escape sequence '\]'
sdk/python/kfp/dsl/for_loop.py:47:30: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/for_loop.py:47:37: W605 invalid escape sequence '\w'
sdk/python/kfp/dsl/for_loop.py:47:44: W605 invalid escape sequence '\['
sdk/python/kfp/dsl/for_loop.py:47:63: W605 invalid escape sequence '\]'
sdk/python/kfp/dsl/for_loop.py:67:17: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/for_loop.py:67:24: W605 invalid escape sequence '\w'
sdk/python/kfp/dsl/for_loop.py:67:31: W605 invalid escape sequence '\['
sdk/python/kfp/dsl/for_loop.py:67:33: W605 invalid escape sequence '\s'
sdk/python/kfp/dsl/for_loop.py:67:39: W605 invalid escape sequence '\w'
sdk/python/kfp/dsl/for_loop.py:67:43: W605 invalid escape sequence '\s'
sdk/python/kfp/dsl/for_loop.py:67:47: W605 invalid escape sequence '\s'
sdk/python/kfp/dsl/for_loop.py:67:68: W605 invalid escape sequence '\]'
sdk/python/kfp/dsl/placeholders_test.py:302:74: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/placeholders_test.py:312:74: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/placeholders_test.py:326:74: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/placeholders_test.py:344:74: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/placeholders_test.py:360:74: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/placeholders_test.py:374:74: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/placeholders_test.py:389:74: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/tasks_group_test.py:77:25: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/tasks_group_test.py:77:50: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/tasks_group_test.py:77:67: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/tasks_group_test.py:77:79: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/types/type_annotations.py:230:30: W605 invalid escape sequence '\.'
sdk/python/kfp/dsl/types/type_annotations.py:230:43: W605 invalid escape sequence '\w'
sdk/python/kfp/dsl/types/type_annotations.py:230:50: W605 invalid escape sequence '\['
sdk/python/kfp/dsl/types/type_annotations.py:230:54: W605 invalid escape sequence '\]'
sdk/python/kfp/local/config_test.py:73:69: W605 invalid escape sequence '\.'
sdk/python/kfp/local/config_test.py:73:76: W605 invalid escape sequence '\.'
sdk/python/kfp/local/config_test.py:73:82: W605 invalid escape sequence '\('
sdk/python/kfp/local/config_test.py:73:84: W605 invalid escape sequence '\)'
sdk/python/kfp/local/config_test.py:73:118: W605 invalid escape sequence '\.'
test/sample-test/sample_test_launcher.py:95:45: W605 invalid escape sequence '\.'
test/sample-test/sample_test_launcher.py:245:17: W605 invalid escape sequence '\.'
test/sample-test/sample_test_launcher.py:245:82: W605 invalid escape sequence '\w'
test/sample-test/sample_test_launcher.py:247:17: W605 invalid escape sequence '\.'
test/sample-test/sample_test_launcher.py:247:69: W605 invalid escape sequence '\w'
test/sample-test/sample_test_launcher.py:252:21: W605 invalid escape sequence '\.'
test/sample-test/sample_test_launcher.py:252:55: W605 invalid escape sequence '\w'
test/sample-test/unittests/utils_tests.py:51:13: W605 invalid escape sequence '\.'
test/sample-test/unittests/utils_tests.py:51:65: W605 invalid escape sequence '\w'
test/sample-test/unittests/utils_tests.py:52:13: W605 invalid escape sequence '\.'
test/sample-test/unittests/utils_tests.py:52:59: W605 invalid escape sequence '\w'
test/sample-test/unittests/utils_tests.py:53:13: W605 invalid escape sequence '\.'
test/sample-test/unittests/utils_tests.py:53:66: W605 invalid escape sequence '\w'
test/sample-test/unittests/utils_tests.py:54:13: W605 invalid escape sequence '\.'
test/sample-test/unittests/utils_tests.py:54:66: W605 invalid escape sequence '\w'

Checklist:

Copy link

Hi @TheKevJames. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@hbelmiro
Copy link
Contributor

/ok-to-test
/rerun-all

@TheKevJames
Copy link
Contributor Author

@hbelmiro any way for me to see the logs of why that test failed? Looks like might be infra issues, but want to double-check it's not a bug in my changes.

@hbelmiro
Copy link
Contributor

@hbelmiro any way for me to see the logs of why that test failed? Looks like might be infra issues, but want to double-check it's not a bug in my changes.

@TheKevJames we only have this. It's failing on several PRs and it seems to be an infra issue indeed.
Let's wait for @chensun's review. He may have an answer for that.

If you want to try it locally, you can try the following. I have it on my notes, but I'm not sure it still works.

source_root=$(pwd)
pip install --upgrade pip
pip install $source_root/sdk/python
pushd api
make clean python
popd
python3 -m pip install api/v2alpha1/python
pip install components/google-cloud
pip install $(grep 'pytest==' sdk/python/requirements-dev.txt)
pytest test/gcpc-tests/run_all_gcpc_modules.py

@TheKevJames
Copy link
Contributor Author

Needed to update your instructions to account for some changes (you need to install protoc, wheel, and fix protobuf's broken packaging), pasted my terminal log below in case you want to update your notes. Test succeeded locally.

/t/pipelines » python3 -m venv venv
/t/pipelines » source venv/bin/activate
(venv) /t/pipelines » pip install --upgrade pip
# SNIP
Successfully installed pip-24.2
(venv) /t/pipelines » pip install ./sdk/python
# SNIP
Successfully installed PyYAML-6.0.2 cachetools-5.5.0 certifi-2024.8.30 charset-normalizer-3.3.2 click-8.1.7 docstring-parser-0.16 google-api-core-2.19.2 google-auth-2.34.0 google-cloud-core-2.4.1 google-cloud-storage-2.18.2 google-crc32c-1.5.0 google-resumable-media-2.7.2 googleapis-common-protos-1.65.0 idna-3.8 kfp-2.8.0 kfp-pipeline-spec-0.3.0 kfp-server-api-2.0.5 kubernetes-26.1.0 oauthlib-3.2.2 proto-plus-1.24.0 protobuf-4.25.4 pyasn1-0.6.0 pyasn1-modules-0.4.0 python-dateutil-2.9.0.post0 requests-2.32.3 requests-oauthlib-2.0.0 requests-toolbelt-0.10.1 rsa-4.9 six-1.16.0 tabulate-0.9.0 urllib3-1.26.20 websocket-client-1.8.0
(venv) /t/pipelines » pushd api
(venv) /t/pipelines » make clean python
# SNIP
protoc is not found.  Please compile it or install the binary package.
(venv) /t/pipelines » curl -sSLo protoc.zip https://github.com/protocolbuffers/protobuf/releases/download/v28.0/protoc-28.0-osx-aarch_64.zip
(venv) /t/pipelines » unzip protoc.zip
(venv) /t/pipelines » export PATH=$(pwd)/bin:$PATH
(venv) /t/pipelines » make clean python
# SNIP
error: invalid command 'bdist_wheel'
(venv) /t/pipelines » pip install wheel
# SNIP
Successfully installed wheel-0.44.0
(venv) /t/pipelines » make clean python
(venv) /t/pipelines » popd
(venv) /t/pipelines » pip install ./api/v2alpha1/python
# SNIP
Successfully installed kfp-pipeline-spec-0.3.0
(venv) /t/pipelines » pip install ./components/google-cloud
# SNIP
      Successfully uninstalled kfp-2.8.0
Successfully installed Jinja2-3.1.4 MarkupSafe-2.1.5 annotated-types-0.7.0 google-cloud-aiplatform-1.64.0 google-cloud-bigquery-3.25.0 google-cloud-pipeline-components-2.16.1 google-cloud-resource-manager-1.12.5 grpc-google-iam-v1-0.13.1 grpcio-1.66.1 grpcio-status-1.62.3 kfp-2.7.0 numpy-2.1.0 packaging-24.1 pydantic-2.8.2 pydantic-core-2.20.1 shapely-2.0.6 typing-extensions-4.12.2
(venv) /t/pipelines » pip install $(grep 'pytest==' ./sdk/python/requirements-dev.txt)
(venv) /t/pipelines » pytest ./test/gcpc-tests/run_all_gcpc_modules.py
# SNIP
ImportError: cannot import name 'runtime_version' from 'google.protobuf'
(venv) /t/pipelines » pip freeze | grep protobuf
protobuf==4.25.4
(venv) /t/pipelines » pip install --upgrade protobuf
# SNIP
Successfully installed protobuf-5.28.0
(venv) /t/pipelines » cp venv/lib/python3.11/site-packages/google/protobuf/runtime_version.py tmp-runtime-version.py
(venv) /t/pipelines » pip install protobuf==4.25.4
# SNIP
Successfully installed protobuf-4.25.4
(venv) /t/pipelines » pytest ./test/gcpc-tests/run_all_gcpc_modules.py
======================================================================================================== 1 passed, 12 warnings in 2.63s =========================================================================================================

@hbelmiro
Copy link
Contributor

Thank you for sharing this @TheKevJames.
I created an issue to migrate this test: #11154

@TheKevJames
Copy link
Contributor Author

Looks like the updated test is in, rebased.

@TheKevJames
Copy link
Contributor Author

@hbelmiro would you please be able to mark this for testing? Thank you!

@hbelmiro
Copy link
Contributor

/ok-to-test
/rerun-all

@TheKevJames
Copy link
Contributor Author

I see the tests have been failing for all commits on mainline as well. I don't think any of these are issues with my code?

@hbelmiro how do y'all prefer to tackle this? I see other PRs are still getting merged despite the failures, not sure what the approach is for that. Otherwise, I guess we can wait until mainline is green and I can rebase again? Please advise.

@hbelmiro
Copy link
Contributor

I see the tests have been failing for all commits on mainline as well. I don't think any of these are issues with my code?

@hbelmiro how do y'all prefer to tackle this? I see other PRs are still getting merged despite the failures, not sure what the approach is for that. Otherwise, I guess we can wait until mainline is green and I can rebase again? Please advise.

@TheKevJames
I confirm we have failing tests. It should not be happening.
We need to fix the master branch ASAP. Once done, we need to rebase this PR.

I see other PRs are still getting merged despite the failures

This should not be happening. There's no point of having tests if we ignore them.
Can you please point the PRs that were merged with failing tests so we can identify the reason and prevent this from happening?

@TheKevJames
Copy link
Contributor Author

Sounds good, please poke me when master is green again and I'd be happy to update!

Looks like "most of them" for the past few days, looking at the mainline commit graph shows every merged PR has failed / continued to fail: https://github.com/kubeflow/pipelines/commits/master/

Looking at the four from today:

@hbelmiro
Copy link
Contributor

hbelmiro commented Sep 17, 2024

@TheKevJames

Sounds good, please poke me when master is green again and I'd be happy to update!

Sure.

Looks like "most of them" for the past few days, looking at the mainline commit graph shows every merged PR has failed / continued to fail: https://github.com/kubeflow/pipelines/commits/master/

There's a slight difference between merging PRs with failing tests and tests failing after PRs get merged.
The first should never happen. The second can happen in some cases and when it happens we should identify and fix ASAP.
What you see in the commit history is the second.

Looking at the four from today:

None of those PRs were merged with failing tests (the first case I mentioned above). We don't run all the tests on PRs. We run only the ones that may be affected by the PR.
See this, for example. These tests are run only when the PR touches one of those files:

      - '.github/workflows/e2e-test.yml'
      - 'scripts/deploy/github/**'
      - 'go.mod'
      - 'go.sum'
      - 'backend/**'
      - 'frontend/**'
      - 'proxy/**'
      - 'manifests/kustomize/**'
      - 'test/**'

@hbelmiro
Copy link
Contributor

@TheKevJames the CI is green again. Can you please rebase?

Signed-off-by: Kevin James <KevinJames@thekev.in>
@TheKevJames
Copy link
Contributor Author

@hbelmiro done!

@hbelmiro
Copy link
Contributor

/ok-to-test
/rerun-all

Copy link
Contributor

@hbelmiro hbelmiro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

cc @chensun @zijianjoy

Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 0b92f86 into kubeflow:master Sep 20, 2024
33 checks passed
zed546213 pushed a commit to zed546213/pipelines that referenced this pull request Sep 23, 2024
Signed-off-by: Kevin James <KevinJames@thekev.in>
Signed-off-by: zed546213 <zed546213@gmail.com>
VaniHaripriya pushed a commit to VaniHaripriya/data-science-pipelines that referenced this pull request Oct 1, 2024
Signed-off-by: Kevin James <KevinJames@thekev.in>
@HumairAK HumairAK added this to the KFP SDK 2.10 milestone Oct 10, 2024
boarder7395 pushed a commit to boarder7395/pipelines that referenced this pull request Oct 17, 2024
Signed-off-by: Kevin James <KevinJames@thekev.in>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants