Skip to content

Commit

Permalink
[SPARK-35791][SQL] Release on-going map properly for NULL-aware ANTI …
Browse files Browse the repository at this point in the history
…join

### What changes were proposed in this pull request?

NULL-aware ANTI join (https://issues.apache.org/jira/browse/SPARK-32290) detects NULL join keys during building the map for `HashedRelation`, and will immediately return `HashedRelationWithAllNullKeys` without taking care of the map built already. Before returning `HashedRelationWithAllNullKeys`, the map needs to be freed properly to save memory and keep memory accounting correctly.

### Why are the changes needed?

Save memory and keep memory accounting correctly for the join query.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing unit tests introduced in apache#29104 .

Closes apache#32939 from c21/free-null-aware.

Authored-by: Cheng Su <chengsu@fb.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit e0d81d9)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
c21 authored and fishcus committed Jan 12, 2022
1 parent 763c27c commit 6ae88cf
Showing 1 changed file with 2 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,7 @@ private[joins] object UnsafeHashedRelation {
// scalastyle:on throwerror
}
} else if (isNullAware) {
binaryMap.free()
return HashedRelationWithAllNullKeys
}
}
Expand Down Expand Up @@ -1064,6 +1065,7 @@ private[joins] object LongHashedRelation {
val key = rowKey.getLong(0)
map.append(key, unsafeRow)
} else if (isNullAware) {
map.free()
return HashedRelationWithAllNullKeys
}
}
Expand Down

0 comments on commit 6ae88cf

Please sign in to comment.