add benchmark of term streams merge

[scampi]

This is a draft PR that adds a benchmark of the term merging operation, to make sure I understood what was expected.
I get a result such as below for the baseline:

test termdict::merger::bench::bench_termmerger_baseline                                         ... bench: 248,879,424 ns/iter (+/- 8,600,537)

Close #469

[fulmicoton]

See inline comments.

[scampi]

Thanks for the comments !
I will address them but I'll create a fst-based term merging prototype before pushing any new commits.

[fulmicoton]

@scampi apologies for the late reaction. I actually wrote the comments 3 weeks ago but forgot to hit the send review button >.<

[fulmicoton]

If you have trouble with rebase, I can do it... Let me know once you addressed the comments

[scampi]

@fulmicoton see commit 27b29d6 with a possible term merger implementation based on the FST union.
Below are some runs of the benchmark. I see some variability in the run, the union approach is not always significantly better than the baseline, although it seems to be faster in general.

# fst union

test termdict::merger::bench::bench_termmerger_baseline                                         ... bench: 257,954,291 ns/iter (+/- 13,201,530)
test termdict::merger::bench::bench_termmerger_baseline                                         ... bench: 234,322,785 ns/iter (+/- 43,848,128)
test termdict::merger::bench::bench_termmerger_baseline                                         ... bench: 341,135,396 ns/iter (+/- 24,285,123)

# baseline

test termdict::merger::bench::bench_termmerger_baseline                                         ... bench: 291,538,403 ns/iter (+/- 54,753,066)
test termdict::merger::bench::bench_termmerger_baseline                                         ... bench: 349,626,129 ns/iter (+/- 78,527,658)
test termdict::merger::bench::bench_termmerger_baseline                                         ... bench: 376,261,843 ns/iter (+/- 15,450,698)

[scampi]

@PSeitz See commit c899c1c with fixes for your review. Thanks!

[PSeitz]

thanks! LGTM

[fulmicoton]

@scampi Thanks it is a very valuable PR for tantivy. Can you add an extra commit to update CHANGELOG.md with your contribution?

[scampi]

@fulmicoton Please see commit 66b633a with changelog update

[fulmicoton]

@scampi Congratulations !