Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: bump spark version to 3.4.3 #292

Merged
merged 3 commits into from
May 29, 2024

Conversation

huaxingao
Copy link
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Copy link
Member

@viirya viirya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only have diff for Spark 3.4.2. To build and test with Spark 3.4.3, you need to prepare diff for Spark 3.4.3.

@huaxingao huaxingao closed this Apr 22, 2024
@huaxingao huaxingao reopened this Apr 22, 2024
@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 34.18%. Comparing base (de8fe45) to head (f6dcdd7).

Additional details and impacted files
@@            Coverage Diff            @@
##               main     #292   +/-   ##
=========================================
  Coverage     34.18%   34.18%           
  Complexity      851      851           
=========================================
  Files           116      116           
  Lines         38570    38570           
  Branches       8531     8531           
=========================================
  Hits          13187    13187           
  Misses        22612    22612           
  Partials       2771     2771           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@parthchandra parthchandra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (pending ci)

Copy link
Contributor

@kazuyukitanimura kazuyukitanimura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

initialSessionOptions: java.util.HashMap[String, String]) = {
this(sc, None, None,
- SparkSession.applyExtensions(
+ SparkSession.applyExtensions(sc,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the most significant change in this PR is this Spark code diff. Do you notice any obvious change to 3.4.2 diffs?

Copy link
Member

@viirya viirya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me as I quickly look through the 3.4.3 diffs. But wondering if @huaxingao finds any changes between 3.4.2 and 3.4.3 diffs we should give notice.

@huaxingao
Copy link
Contributor Author

@viirya
I didn't see any changes between 3.4.2.diff and 3.4.3.diff except some index changes, line number changes and the changes I added

@@ -1128,7 +1130,7 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
             .where(s"a < ${Long.MaxValue}")
             .collect()
         }
-        assert(exception.getCause.getCause.isInstanceOf[SchemaColumnConvertNotSupportedException])
+        assert(exception.getMessage.contains("Column: [a], Expected: bigint, Found: INT32"))
       }
     }
   }

@viirya
Copy link
Member

viirya commented May 29, 2024

@viirya I didn't see any changes between 3.4.2.diff and 3.4.3.diff except some index changes, line number changes and the changes I added

Okay, that's great. I thought that there might be some conflicts between 3.4.2 and 3.4.3 tests, but seems we don't have much changes between the two patch versions.

@huaxingao huaxingao merged commit 8f8a0d9 into apache:main May 29, 2024
40 checks passed
@huaxingao
Copy link
Contributor Author

Thanks @kazuyukitanimura @parthchandra @viirya

@huaxingao huaxingao deleted the bump_spark_version branch May 29, 2024 06:18
himadripal pushed a commit to himadripal/datafusion-comet that referenced this pull request Sep 7, 2024
* build: bump spark version to 3.4.3

* add 3.4.4 diff

* fix 3.4.3.diff

---------

Co-authored-by: Huaxin Gao <huaxin.gao@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants