Spark: Allow importing empty tables #7980
Conversation
| importSparkPartitions( | ||
| spark, sourceTablePartitions, targetTable, spec, stagingDir, checkDuplicateFiles); | ||
| if (sourceTablePartitions.isEmpty()) { | ||
| targetTable.newAppend().commit(); |
There was a problem hiding this comment.
What's the reason of empty commit, instead of say, returning ?
There was a problem hiding this comment.
That's what we do for un-partitioned table. And we expect the migration produces a snapshot so that we know the number of migrated files: https://github.com/apache/iceberg/blob/master/spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/MigrateTableSparkAction.java#L159
|
Makes sense, Ill wait to see if any comments, will commit if not. |
|
@szehon-ho Thanks for reviewing. Do I need to port the change to other spark versions? |
|
Merged, thanks @lirui-apache , yea I think that would be great , maybe at least 3.3? I think It's voluntary and depends who needs what in each branch. |
|
Would love to see this ported to 3.2 as well, thanks! Otherwise I can do it. |
|
Sure I'll open a PR for other spark versions. |
To address #7929.
When a partitioned table is empty, create an empty append commit instead of throwing an exception.