Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Don't Review] Only for list python packages for branch-3.4 #45600

Closed

Conversation

panbingkun
Copy link
Contributor

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

@github-actions github-actions bot added the INFRA label Mar 20, 2024
dongjoon-hyun pushed a commit that referenced this pull request Mar 20, 2024
… failure

### What changes were proposed in this pull request?
The pr aims to fix `python linter issue` on branch-3.4 through pinning `matplotlib<3.3.0`

### Why are the changes needed?
- Through this PR #45600, we found that the version of `matplotlib` in our Docker image was `3.8.2`, which clearly did not meet the original requirements for `branch-3.4`.
  https://github.com/panbingkun/spark/actions/runs/8354370179/job/22869580038
  <img width="1072" alt="image" src="https://github.com/apache/spark/assets/15246973/dd425bfb-ce5f-4a99-a487-a462d6ebbbb9">
  https://github.com/apache/spark/blob/branch-3.4/dev/requirements.txt#L12
  <img width="973" alt="image" src="https://github.com/apache/spark/assets/15246973/70485648-b886-4218-bb21-c41a85d5eecf">

- Fix as follows:
<img width="989" alt="image" src="https://github.com/apache/spark/assets/15246973/db31d8fb-0b6c-4925-95e1-0ca0247bb9f5">

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #45608 from panbingkun/branch_3.4_pin_matplotlib.

Authored-by: panbingkun <panbingkun@baidu.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@panbingkun
Copy link
Contributor Author

Fo record:

1.success

(base) panbingkun:~/Developer/spark/spark-trunk$pip install --upgrade --no-cache-dir --force-reinstall ./python/dist/pyspark-3.4.3.dev0.tar.gz
Processing ./python/dist/pyspark-3.4.3.dev0.tar.gz
  Preparing metadata (setup.py) ... done
Collecting py4j==0.10.9.7 (from pyspark==3.4.3.dev0)
  Downloading py4j-0.10.9.7-py2.py3-none-any.whl.metadata (1.5 kB)
Downloading py4j-0.10.9.7-py2.py3-none-any.whl (200 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.5/200.5 kB 134.2 kB/s eta 0:00:00
Building wheels for collected packages: pyspark
  Building wheel for pyspark (setup.py) ... done
  Created wheel for pyspark: filename=pyspark-3.4.3.dev0-py2.py3-none-any.whl size=626013326 sha256=e423146045a43d8c90ac471489dd5b6fb3702def94274f6db968961c8981731f
  Stored in directory: /private/var/folders/kl/p0fxfzp53g5cc38qjxqwk_cm0000gp/T/pip-ephem-wheel-cache-73zlf_ix/wheels/0d/86/cd/947463e4be802583aec12b9dd00109a2a65ea91253d3ffd32d
Successfully built pyspark
Installing collected packages: py4j, pyspark
  Attempting uninstall: pyspark
    Found existing installation: pyspark 4.0.0.dev0
    Uninstalling pyspark-4.0.0.dev0:
      Successfully uninstalled pyspark-4.0.0.dev0
Successfully installed py4j-0.10.9.7 pyspark-3.4.3.dev0

2.fail

Writing pyspark-3.4.3.dev0/setup.cfg
creating dist
Creating tar archive
removing 'pyspark-3.4.3.dev0' (and everything under it)
Installing dist into virtual env
Processing ./python/dist/pyspark-3.4.3.dev0.tar.gz
  Preparing metadata (setup.py) ... 25l-� �done
25hCollecting py4j==0.10.9.7 (from pyspark==3.4.3.dev0)
  Downloading py4j-0.10.9.7-py2.py3-none-any.whl.metadata (1.5 kB)
Downloading py4j-0.10.9.7-py2.py3-none-any.whl (200 kB)
25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/200.5 kB ? eta -:--:--
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.5/200.5 kB 9.1 MB/s eta 0:00:00
25hBuilding wheels for collected packages: pyspark

@dongjoon-hyun
Copy link
Member

Thank you for investigation. Ya, it's really weird flakiness.

@panbingkun
Copy link
Contributor Author

Thank you for investigation. Ya, it's really weird flakiness.

@dongjoon-hyun
The above problem has been addressed, and a detailed explanation is provided here: #45624

@dongjoon-hyun
Copy link
Member

Then, shall we close this PR, @panbingkun ?

@panbingkun panbingkun closed this Mar 24, 2024
szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Aug 7, 2024
… failure

### What changes were proposed in this pull request?
The pr aims to fix `python linter issue` on branch-3.4 through pinning `matplotlib<3.3.0`

### Why are the changes needed?
- Through this PR apache#45600, we found that the version of `matplotlib` in our Docker image was `3.8.2`, which clearly did not meet the original requirements for `branch-3.4`.
  https://github.com/panbingkun/spark/actions/runs/8354370179/job/22869580038
  <img width="1072" alt="image" src="https://github.com/apache/spark/assets/15246973/dd425bfb-ce5f-4a99-a487-a462d6ebbbb9">
  https://github.com/apache/spark/blob/branch-3.4/dev/requirements.txt#L12
  <img width="973" alt="image" src="https://github.com/apache/spark/assets/15246973/70485648-b886-4218-bb21-c41a85d5eecf">

- Fix as follows:
<img width="989" alt="image" src="https://github.com/apache/spark/assets/15246973/db31d8fb-0b6c-4925-95e1-0ca0247bb9f5">

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#45608 from panbingkun/branch_3.4_pin_matplotlib.

Authored-by: panbingkun <panbingkun@baidu.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants