- Expose internal API to get Spark version on the driver side (#429)
- Add
SPARK_VERSION_OVERRIDE
environment variable (#432)
- None
The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:
Spark Version | microsoft-spark JAR |
---|---|
2.3.* | microsoft-spark-2.3.x-0.10.0.jar |
2.4.0 | microsoft-spark-2.4.x-0.10.0.jar |
2.4.1 | |
2.4.3 | |
2.4.4 | |
2.4.5 | |
2.4.2 | Not supported |