RUMM-3338: Introduce known file cache and cleanup throttling in BatchFileOrchestrator in order to reduce the number of syscalls

[0xnm]

What does this PR do?

The goal of this PR is to reduce CPU consumption during the scenarios like described in #1469.

The issue comes from the fact that when we have many batch files on the disk calling listSortedBatchFiles as-is (which is called several times during getWritableFile, for example) becomes quite expensive because:

• syscalls (like isFile) are expensive
• pattern matching becomes expensive as well.

In order to optimize that, the following is done:

• Introduced the cache for the files already known to be batch files, so that we don't need to call isFile and use regex.
• Introduce throttling for the cleanup (only for the getWritableFile path, because a) it can be called much more often than getReadableFile, for every event; b) cleanup actions are a bit different between getReadableFile and getWritableFile, so having the same timestamp is not quite correct).

Test scenario: 300 batch files on the disk, disabled network, writing 1 log entry every 100 ms.

CPU usage after the change (same scenario and duration is used):

It gives a very good bump if the total number files on the disk is below the cache max size. If cache is not enough, the problem comes back (but maybe in a bit better state, because we have cleanup throttling)

CPU usage before the change:

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Make sure you discussed the feature or bugfix with the maintaining team in an Issue
• Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

[mariusc83]

Nice job ;)

[codecov-commenter]

Codecov Report

Merging #1506 (9538a69) into develop (a982b35) will increase coverage by 0.07%.
The diff coverage is 90.00%.

@@             Coverage Diff             @@
##           develop    #1506      +/-   ##
===========================================
+ Coverage    82.00%   82.07%   +0.07%     
===========================================
  Files          363      363              
  Lines        12981    12995      +14     
  Branches      2162     2165       +3     
===========================================
+ Hits         10645    10665      +20     
+ Misses        1679     1676       -3     
+ Partials       657      654       -3     

Impacted Files	Coverage Δ
...al/persistence/file/batch/BatchFileOrchestrator.kt	95.00% <88.89%> (+1.30%)	⬆️
...internal/persistence/file/FilePersistenceConfig.kt	88.89% <100.00%> (+1.39%)	⬆️

... and 9 files with indirect coverage changes

[xgouchet]

maybe the update of lastCleanupTimestamp can be part of the deleteObsoleteFiles() method?