Skip to content

Commit

Permalink
[SPARK-35791][SQL] Release on-going map properly for NULL-aware ANTI …
Browse files Browse the repository at this point in the history
…join

### What changes were proposed in this pull request?

NULL-aware ANTI join (https://issues.apache.org/jira/browse/SPARK-32290) detects NULL join keys during building the map for `HashedRelation`, and will immediately return `HashedRelationWithAllNullKeys` without taking care of the map built already. Before returning `HashedRelationWithAllNullKeys`, the map needs to be freed properly to save memory and keep memory accounting correctly.

### Why are the changes needed?

Save memory and keep memory accounting correctly for the join query.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing unit tests introduced in #29104 .

Closes #32939 from c21/free-null-aware.

Authored-by: Cheng Su <chengsu@fb.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
c21 authored and cloud-fan committed Jun 17, 2021
1 parent 947c7ea commit e0d81d9
Showing 1 changed file with 2 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,7 @@ private[joins] object UnsafeHashedRelation {
throw QueryExecutionErrors.cannotAcquireMemoryToBuildUnsafeHashedRelationError()
}
} else if (isNullAware) {
binaryMap.free()
return HashedRelationWithAllNullKeys
}
}
Expand Down Expand Up @@ -1060,6 +1061,7 @@ private[joins] object LongHashedRelation {
val key = rowKey.getLong(0)
map.append(key, unsafeRow)
} else if (isNullAware) {
map.free()
return HashedRelationWithAllNullKeys
}
}
Expand Down

0 comments on commit e0d81d9

Please sign in to comment.