LogMergePolicy set_max_merge_size not being respected

[andrebsguedes]

Describe the bug

• What did you do?
  Configured the LogMergePolicy with max merge size and inserted 38M documents with a single final commit in the end.

let writer = index
    .writer(1024 * 1024 * 1024)
    .expect("failed to create writer");

let mut policy = tantivy::merge_policy::LogMergePolicy::default();
policy.set_max_merge_size(3_000_000);

writer.set_merge_policy(Box::new(policy));

• What happened?

Multiple 1M document segments merged into 9M document segments and two 9M document segments merged into a ~ 18M document segment

• What was expected?

The expected behavior was that no segment larger than 3M documents would be merged. As an aside, the desired behavior for my use case was to limit the maximum size of a segment to 3M documents in order to have enough segments to optimize parallelism.

Which version of tantivy are you using?
If "master", ideally give the specific sha1 revision.

Version 13.2

To Reproduce

If your bug is deterministic, can you give a minimal reproducing code?
Some bugs are not deterministic. Can you describe with precision in which context it happened?
If this is possible, can you share your code?

The time required to reindex all documents is not trivial so no attempt was made to reproduce the bug

[fulmicoton]

@andrebsguedes Thank you for the bug report... We will investigate.

[fulmicoton]

@PSeitz can you have a look at this?

[PSeitz]

Thanks for the report, should be fixed with #1035