feat(Build): Optimize Build Examples Test

[Jake-Carter]

Description

Closes #1121

The .github/workflows/scripts/build.py script now accepts a --change_files optional argument.

  --change_file CHANGE_FILE
                        (Optional) Pass a text file containing a list of space-separated or new-line separated changed files. The build
                        script will intelligently adjust which parts it tests based on this list.

It takes about 40s to generate the dependency map on the Github runners.

Now, if a change is made to Cordio only the BLE micros will be re-tested. If a change is made to only one micro's examples, the other micros will not be fully re-tested either.

Also optimized the number of clean.cordio calls. A special case is still needed (see cd1bf5b) but this is a big help.

Overall this should speed up the workflow. Full run 47m -> 29m, and should reduce unnecessary full runs.

TODO

• The granularity of the dependency map can be improved in the future. This first implementation is not meXX or rev_XX-aware, so a change in PeriphDrivers will rebuild everything.

• Dependencies could also be saved per example so that only relevant examples are tested. There will be some tradeoff in the script complexity and dependency map creation time for this.

• Clean up the watch files step to use the new change file action

[Jake-Carter]

Will need to deal with forks

[EricB-ADI]

Wasn't aware of this side effect. git add remote parent https://github.com/analogdevicesinc/msdk.git git diff parent/main --name-only is the easy way to see what changed between this and main. That is independent of the fork. Gonna need to disable errors around `git add remote`` otherwise you get an error about remote already existing in the case it is not a fork.

Can add a grep to the git diff to look for the different types. and push them into the correct jobs too.

[Jake-Carter]

@EricB-ADI looks like it just needed a rebase, I'd already had the workflow checking out ${{ github.event.pull_request.head.repo.full_name }}

I updated the build system to pass through make query requests to sub-makes that build the library files. This let's us generate a 1:1 dependency map (i.e. Libraries/PeriphDrivers/Source/GPIO/gpio_ai85.c will only re-test AI85).

CFS relies on make query, though, so I'll need to check if this breaks their parser.

If you can check out this branch and make some tests that'd be helpful to see if I missed any edge cases. This will test any local modifications

git diff --name-only > change_files
python .github/workflows/scripts/build.py --change_file change_files

[EricB-ADI]

Are we happy with the speedup or do we want to go further?

[Jake-Carter]

Are we happy with the speedup or do we want to go further?

@EricB-ADI I think I'm happy with these initial results for small incremental changes, which will now only re-test micros that are directly dependent. In those cases we go from 47m -> ~5m.

We could definitely still speed up the full runs following your suggestion to parallelize the job into separate jobs for each micro. The script currently supports that mechanism with the --targets option.

The work would mainly be on the Github Actions side to break the job out into separate conditional stages.

I think I'd prefer to start with this first pass to see how much of an improvement we get in practice and to stress test the dependency mapping.