Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-26607][python] There are multiple MAX_LONG_VALUE value errors in pyflink code #19089

Closed
wants to merge 6 commits into from

Conversation

cun8cun8
Copy link
Contributor

What is the purpose of the change

This PR is mainly to repair MAX_LONG_VALUE value
MAX_LONG_VALUE = sys.maxsize

maxsize attribute of the sys module fetches the largest value a variable of data type Py_ssize_t ** can store. It is the Python platform’s pointer that dictates the maximum size of lists and strings in Python. The size value returned by maxsize depends on the platform architecture:

32-bit: the value will be 2^31 – 1, i.e. 2147483647
64-bit: the value will be 2^63 – 1, i.e. 9223372036854775807

Brief change log

"""
A constant holding the maximum value a long can have, 2^63 – 1.
"""
MAX_LONG_VALUE = 0x7fffffffffffffff

Verifying this change

Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (100MB)
  • Extended integration test for recovery after master (JobManager) failure
  • Added test that validates that TaskInfo is transferred only once across recoveries
  • Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@cun8cun8 cun8cun8 changed the title [FLINK-26607][python] Fix MAX_LONG_VALUE value [FLINK-26607][python] There are multiple MAX_LONG_VALUE value errors in pyflink code Mar 15, 2022
@flinkbot
Copy link
Collaborator

flinkbot commented Mar 15, 2022

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@cun8cun8 cun8cun8 force-pushed the branch1 branch 2 times, most recently from 103331d to 146e605 Compare March 15, 2022 09:37
Copy link
Contributor

@dianfu dianfu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cun8cun8 Thanks for the contribution. Have left a minor comments. Besides, please also update other places, such as window_operator.py, window_aggregate_fast.pyx.

@cun8cun8 cun8cun8 force-pushed the branch1 branch 2 times, most recently from 6c2a22d to d26c5ac Compare March 16, 2022 10:19
@dianfu dianfu closed this in d2d0065 Mar 17, 2022
dianfu pushed a commit to dianfu/flink that referenced this pull request Mar 17, 2022
dianfu pushed a commit that referenced this pull request Mar 17, 2022
JasonLeeCoding pushed a commit to JasonLeeCoding/flink that referenced this pull request May 27, 2022
zstraw pushed a commit to zstraw/flink that referenced this pull request Jul 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants