Skip to content

Commit

Permalink
[SPARK-46087][PYTHON] Sync PySpark dependencies in docs and dev requi…
Browse files Browse the repository at this point in the history
…rements

### What changes were proposed in this pull request?
This PR proposes to synchronize the versions of dependencies listed in the [PySpark documentation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies) with those specified in the [dev/requirements.txt](https://github.com/apache/spark/blob/master/dev/requirements.txt) file.

### Why are the changes needed?

Aligning the versions of dependencies ensures that the development environment reflects the actual user environment more accurately.

### Does this PR introduce _any_ user-facing change?

No API changes.

### How was this patch tested?

Build the documents from latest master branch manually and sync the version of dependencies:

<img width="774" alt="Screenshot 2023-11-24 at 2 49 09 PM" src="https://github.com/apache/spark/assets/44108233/4f539fc7-0bbc-4fe3-949f-b4c225122b56">

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#44000 from itholic/req-sync.

Authored-by: Haejoon Lee <haejoon.lee@databricks.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information
itholic authored and dongjoon-hyun committed Nov 24, 2023
1 parent 4530b12 commit 441e15e
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions dev/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# PySpark dependencies (required)
py4j
py4j>=0.10.9.7

# PySpark dependencies (optional)
numpy
pyarrow
numpy>=1.21
pyarrow>=4.0.0
six==1.16.0
pandas
pandas>=1.4.4
scipy
plotly
mlflow>=2.3.1
Expand Down Expand Up @@ -52,8 +52,8 @@ black==23.9.1
py

# Spark Connect (required)
grpcio==1.59.3
grpcio-status==1.59.3
grpcio>=1.59.3
grpcio-status>=1.59.3
protobuf==4.25.1
googleapis-common-protos>=1.56.4

Expand Down

0 comments on commit 441e15e

Please sign in to comment.