pkg/stm32cmsis: retrieve STM32 CMSIS header from a package

[aabadie]

Contribution description

This PR is a POC that adds stm32cube repositories as a single external package (example: the F4 repository). Each stm32cube repository contains all the CMSIS headers of each STM32 families (for F4 it's in "Drivers/CMSIS/Device/ST/STM32F4xx/Include").

The stm32cube package is only used for fetching the headers automatically when building an application targetting an STM32 microcontroller. Thus, this PR also removes all STM32 vendor headers from the codebase.

Not everything is perfect, there are pros and cons:

• pros:

  • updating the CMSIS headers for a given STM32 family can be done by simply changing the corresponding commit hash of the package. All CMSIS headers are kept in sync.
  • all headers are available, so there's no need to add new ones for each new STM32 cpu support added
  • they are kept unchanged, so easier to maintain
  • if we could have a more generic solution for vectors and IRQs, this opens the door for the basic support of any cpu model of an already supported family
  • the changeset of this PR is fun: 68 additions and 708,552 deletions.

• cons:

  • a network connection is required for building any application, for a given STM32, the first time, even for a simple hello-world application
  • some versions of stm32cube are huge (hundreds of MB). This is the case for F4, F7, L4. Because the repository also contains a big bunch of blobs (sample audio files, video, etc) which are used as samples in example applications. Fetching the package can take ages and I have no idea of the disk usage increase on Murdock.

I think the pros are very important and here are some ideas I have to mitigate a bit the cons. All of them try to fetch the stm32cube only once in a single location. This way, only the very first build of an application will fetch the stm32cube code (for a given family), the next builds will just reuse the already downloaded headers.

• add some logic to the package to copy the headers from the package location (in bin/pkg/...) to cpu/stm32/include/vendor. The package will also contains some logic to prevent the fetch if the vendors headers are already there or maybe if a lock file is already there and contains the same version.
• add this specific download logic in cpu/stm32/stm32cube.inc.mk and don't use the usual package mechanism.

There might be other solutions I haven't thought of so any suggestion is very welcome.

If you think that this PR is useless or that the cons are too much a pain. Then we can close this and continue the usual STM32 way.

Testing procedure

• A green Murdock
• Verify that the total Murdock jobs time doesn't increase too much

Issues/PRs references

Based on #14144

[benpicco]

My worry is that this is pulled in for every application.
We can do better though: the tools in dist/tools are only downloaded once and then used for all applications.

Now an easy hack would be to make stm32cube a 'tool', but maybe we can add a global package cache (i.e. don't store them in bin/ but in some common folder) instead that can be used for all packages.

[aabadie]

My worry is that this is pulled in for every application.

Exactly what I said but less clear.

Now an easy hack would be to make stm32cube a 'tool'

Sound reasonable. I can try that approach.

we can add a global package cache (i.e. don't store them in bin/ but in some common folder) instead that can be used for all packages.

I think that this would make the most sense but that might not work for the packages built with their own build system (openthread and maybe some others).

[vincent-d]

I like the idea, it will be much easier to maintain and add new cpu/boards.

I think a global package cache would be great for all packages.

Another solution might be to use git submodules, so it's downloaded only once, globally. I'm not sure if it's possible though.

[aabadie]

I think a global package cache would be great for all packages.

Using the PKG_BUILDDIR variable does the job quite well: all stm32cube repositories are cloned under pkg/stm32cube/cache/<cpu family> and the package is only fetched during the first build of a given family. So only once per family.

[kaspar030]

+84 −708,552 😄

[aabadie]

It seems like Murdock doesn't like it :)

[aabadie]

I just pushed a major update of this PR: now each repo is not targetting STM32Cube<cpu fam uppercase> but cmsis_device_<cpu_fam>. The latter is a sub part of the Cube that only contains the CMSIS header files plus other small system files. It's thus much smaller and faster to fetch (no comparison with the previous version). The path to the header files is also less deep in each repository.

I'll have to edit the commit messages when I'll squash to rename stm32cube to stm32cmsis there as well.

Regarding the caching directory, I kept it even if the download is faster because I still think it's handy to not clone the repo all the time. It's disabled on the CI because of incompatibilities with Murdock.

[benpicco]

I like this a lot!
Tried to build some boards locally and it all went smooth.

Let's see what Murdock thinks of it when it's back to normal operation.

[kaspar030]

Let's see what Murdock thinks of it when it's back to normal operation.

This adds ~1.5g (from ~0.5g) to git-cache, making git hit scaling issues. please don't build for now!

[benpicco]

I guess we could just use stm32f030x6.h instead - I don't rember where I got that stm32f030x4.h file from - the only source on Google is 404

[aabadie]

we could just use stm32f030x6.h instead - I don't rember where I got that stm32f030x4.h

That would make my life easier indeed :) Do you have that board for testing ?

[aabadie]

I don't rember where I got that stm32f030x4.h file from

I think this CPU has existed (it's mentioned on the ST website) so probably it could be in an old version of the STM32 Cube. I'll add a specific handling for this one in the patch

[benpicco]

Do you have that board for testing?

Yea, at that price I couldn't pass on it :)
Although it's not very useful with just 16k Flash. Maybe it will get a bit more useful with picolibc.

[aabadie]

Yea, at that price I couldn't pass on it :)

Off-topic: they propose other interesting platforms (8bit STMs :) )

[aabadie]

I triggered this one twice on Murdock and got no failures. It seems to be ~2 minutes slower.

Some detailed numbers:

• last run: total cpu runtime: 3d:23h:28m:31.5s
• previous run: total cpu runtime: 3d:14h:18m:21.9s

So in total, it adds 9h10m.

[kaspar030]

So in total, it adds 9h10m.

offline discussion: seems like a lot. I'll run some benchmarks to pinpoint where that comes from.

[kaspar030]

So I ran an isolated Murdock run (on breeze, a 16core Ryzen box) building for iotlab-m3 and nucleo-f401re, this PR and its merge base. I built each twice, one time with cleared ccache, then again.

It looks good. the cold cache build (which probably triggered an initial download of the necessary headers into git-cache, which blocks other cores that need the same) took 5:37 vs 5:17 minutes pr/master, with hot caches (and the header repos in git-cache) took 2:52 vs 2:47, which is within measurement accuracy.

So let's go, I guess we won't miss those 700k lines much :)

Nice job @aabadie!

[kaspar030]

ACK.

[kaspar030]

For reference, here's the output of the test runs:

master: https://gist.github.com/81dd9278aff04fcfc0d4b3b6b7ffe749
this pr: https://gist.github.com/589574be7962b33083e563b8a2cc1b54