Optimize merge algorithm for largest data sizes

[SergeyKopienko]

In this PR we optimize merge algorithm for data sizes equal or greater then 8M items (16M items for int type).

The main idea - we doing two submits:

1. in the first submit we find split point in some "base" diagonal's subset.
2. in the second submit we find split points in all other diagonal and run serial merge for each diagonal (as before).
   But when we find split point on the current diagonal, we setup some indexes limits for rng1 and rng2: by this way we decrease the amount of reading data from source data ranges. For these limits we load split point's data from previous and next "base" diagonals, calculated on the step (1).

Applying this approach we have good perf profit for biggest data sizes with float and int data types.

Details:

Probably I should explain why I have create

struct __result_and_scratch_storage_base
{
    virtual ~__result_and_scratch_storage_base() = default;
};

and why I inherit __result_and_scratch_storage from __result_and_scratch_storage_base :

template <typename _ExecutionPolicy, typename _T>
struct __result_and_scratch_storage : __result_and_scratch_storage_base
{
    // ...
};

Let me explain the reason.
In the __parallel_merge(oneapi::dpl::__internal::__device_backend_tag, ...) function we checks in run-time some conditions to make decision which submitter we will call:

1. __parallel_merge_submitter<std::uint32_t, ...>
2. __parallel_merge_submitter_large<std::uint32_t, ...>
3. __parallel_merge_submitter_large<std::uint64_t, ...>

In the cases (2) and (3) we should extend the life-time of different __result_and_scratch_storage instances which were created inside __result_and_scratch_storage<std::uint32_t>::operator() or __result_and_scratch_storage<std::uint64_t>::operator():

• in the case (1) we will not have any __result_and_scratch_storage;
• in the case (2) we will have __result_and_scratch_storage<std::uint32_t>;
• in the case (3) we will have __result_and_scratch_storage<std::uint64_t>.

So the std::shared_ptr<__result_and_scratch_storage_base> is some common interface for the both of them which can extend their life-time.

About virtual ~__result_and_scratch_storage_base() = default - it's needed to properly destroy instances of inherited classes, which are really owned by shared-pointer instance in the cases (2) and (3). Without virtual here we will have memory-leaks.

[SergeyKopienko]

Other than the comment about __find_start_point, I think this is in good shape and I have no other comments. Hopefully we can find resolution there, and without loss of performance.

I have fixed two moments in the __find_start_point_in function and delete __find_start_point and it's usages at all.
My checks shown me that there is no performance degradation at this step. only profit.

So I propose to save __find_start_point_in only like in mt last commit.

May be make sense only one question here: do we prefer to rename __find_start_point_in to __find_start_point or save as is?

[mmichel11]

LGTM. Great optimization work! Please wait for the final reviews from others as well.

[danhoeflinger]

LGTM.
I think it worked out nicely.
As mentioned by @mmichel11, I still think there are couple places we have room for more optimization with more effort but this is a big step in the right direction.

Thanks @SergeyKopienko !

[MikeDvorskiy]

? : operator is used instead of if.. else in my proposal. It is by performance reasons? I've just interesting in...
(if.. else was more readable, IMHO)

[SergeyKopienko]

The implementation of __serial_merge on ternary operator gave us good profit. So I hope some small profit we may have in this place too. But I am not ready to prepare any precision data about this.

[MikeDvorskiy]

I see.

[MikeDvorskiy]

LGTM