feat(performance): reduce the number of files scanned after merge conflict

[fnareoh]

Context

The goal of this Merge Request is to improve the speed of the ggshield pre-commit scan when commiting after a merge conflict.

What has been done

I created a script gen_merge_commit.py (temporarily added at the repository root) that creates a repo with two branches (master and feature_branch) a merge commit (with or without commit depending on the options) giving the following commit graph:

*   17d3a27 (HEAD -> feature_branch) Merge branch 'master' into feature_branch
|\  
| * db20e84 (master) Commit master n°2
| * 53a4cd0 Commit master n°1
| * a78f68d Commit master n°0
* | 9f5717c Commit feature branch
|/  
* e1db622 Initial commit

I then experimented and with it:

• Without conflict: our pre-commit does not seem to be called on merge
• With conflict: I implemented a proof of concept way to limit the number of files scan: when ggshield pre-commit is called
  - if MERGE_HEAD does not exists, we assume it is a regular commit and we should scan everything (current behavior)
  - if MERGE_HEAD exists: we list the files about to be committed (git diff --staged --name-only), then compare the current shas of those files (using the command git ls-files --staged <list_files_diff>) with the shas of HEAD and MERGE_HEAD (obtained via git ls-tree <list_files_diff>), and if it differs from both, then the file has been modified and should be re-scanned...

I'm opening this PR as a draft as it is clearly not ready (no proper automated testing) but the general idea should be reviewed...

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.54%. Comparing base (40a904b) to head (c3b3b62).
Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #946      +/-   ##
==========================================
+ Coverage   91.49%   91.54%   +0.05%     
==========================================
  Files         178      178              
  Lines        7321     7372      +51     
==========================================
+ Hits         6698     6749      +51     
  Misses        623      623              

Flag	Coverage Δ
unittests	91.54% <100.00%> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[agateau-gg]

Looks good overall, your approach makes sense.

I am wondering if we always want this behavior or if it should be opt-in 🤔. What do you think?

One remark: calls to ls-files and ls-tree should use -z to handle files with what git calls "unusual names" (*). An easy way to create an "unusual name" is to create one with a double quote in it.

(*): Other parts of ggshield would need to be fixed with regard to that too...

[fnareoh]

@agateau-gg I started doing something because it looked convenient in my tests and possibly useful but maybe you won't like it: I added an option to ggshield install to install the hook with option.

For example, ggshield install -o --json will install the following hook: ggshield secret scan pre-commit --json "$@".

But as it seems to fail on windows, I would like to check with you before debugging... 😓

[agateau-gg]

@agateau-gg I started doing something because it looked convenient in my tests and possibly useful but maybe you won't like it: I added an option to ggshield install to install the hook with option.

For example, ggshield install -o --json will install the following hook: ggshield secret scan pre-commit --json "$@".

But as it seems to fail on windows, I would like to check with you before debugging... 😓

The idea sounds good, but it should be in a separate commit or maybe even in a separate PR.

[agateau-gg]

Some minor remarks, but this is much easier to read!