Combine low seletivity vectors generated by hash join filter

[zhli1142015]

Combine low seletivity vectors generated by join filter
Problem
We found in TPCDS query72 that the join filter leads to a large number of
low-seletivity result vectors, which affects the performance of subsequent
operations.

Details:
The number of input vectors on the probe side of the corresponding join is
84,087, with a total of 743,851,486 rows (our batch size is set to 10,240). Due
to a large number of duplicate rows on the build side, the final result
inflates. The number of output vectors from the join is 2,806,054. The
corresponding join filter filters out a large portion of the results, so the
number of output rows is 1,430,253,235. This leads to the output of many
sparse vectors (the average batch row count is 504).

Solution
The original logic continues the loop to fill more rows if no rows pass the
filters. To resolve this issue, we can extend it to handle cases where only
partial rows pass the filters. We need to ensure that the indices
'outputRowMapping_' and 'outputTableRows_' are filled as much as possible until
we either reach the preferred batch size or have processed all rows in the
current input vector.

This approach will not only address the issue mentioned above but also avoid
unnecessary data copying.

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	c691994
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/6708e64aab82d60008924b91

[zhli1142015]

In our experiment for TPCDS query72, we observed that due to the reduced number of vectors, there is a noticeable performance improvement in operators after the first join and the query latency improves 45% also(57s --> 31s).

Before this change
Join generates low seletivity vectors ( probe input vectors: 84087 --> output vectors: 2806054)

Op after above Join

With this changer
probe input vectors: 84087 --> output vectors: 324659

Op after above Join: the total probe time reduced from 2.5 min to 1.1 min

[zhli1142015]

@mbasmanova and @Yuhta , could you help to take a look?
Thanks.

[zhli1142015]

CI failure is caused by #10871, thanks.

[Yuhta]

I think the idea is great, leave some initial comments.

Related issue #7801

[zhli1142015]

Hello @Yuhta and @mbasmanova , could you take a look again? Thanks.

[Yuhta]

We can remove this

[Yuhta]

Can we do leftSemiProjectIsNull_.resize(outputTableRowsCapacity_) here so we don't need to do it in each loop? SelectivityVector::resize requires iterate over all the bits in it.

[zhli1142015]

Updated, thanks.

[zhli1142015]

CI failure is not related to this PR, thanks.