[ffmpeg] building ffmpeg takes 4 hours on windows (github actions)

[Kolfering]

We have a C++ project using vcpkg with manifest mode to manage the dependencies (all static linked) for mac & windows, and one of those is ffmpeg (with limited features). We use github actions as CI and building ffmpeg on windows (for 32 & 64 bits) started taking an huge amount of time 3-4 weeks ago. It used to be between 20 and 40 minutes but it's now up to 4 hours.

Here is our vcpkg.json file: https://github.com/Aleph-One-Marathon/alephone/blob/master/vcpkg.json

Nothing changed on our side, we are still using the 5.0.2#3 ffmpeg version and our vcpkg.json didn't have to change for the problem to occur, only the vcpkg version from the runner.
However I can't reproduce the issue locally where the build time is still the good one, it seems to only happens in github actions, windows runners, which is why I first filled an issue in their repo but they asked me to come here first. (actions/runner-images#9085)
I don't know if upgrading ffmpeg version would fix it, since it's not an option for us right now anyway. (edit: it doesn't)

Based on the vcpkg version installed in the runners, I can say everything was working fine with vcpkg built from commit cd5e746. (Image Version: 20231126.1.0) (exemple: https://github.com/Aleph-One-Marathon/alephone/actions/runs/7109261671/job/19353897438)

While the problem appears with vcpkg built from commit 08c4e71. (Image Version: 20231205.1.0)
Here are some github actions runs showing the problem:

• https://github.com/Aleph-One-Marathon/alephone/actions/runs/7143256800
• https://github.com/Aleph-One-Marathon/alephone/actions/runs/7361510736

This is not a random behavior, every builds are taking ~ 4 hours without exception since the problem occured.

Environment

• OS: Windows

Expected behavior

building ffmpeg should take ~ 20 mins.

Failure logs

• Here are the logs of ffmpeg built on windows (32 bits) github actions when it's taking 4 hours. I didn't see anything suspect but I may have missed something obvious. https://github.com/Kolfering/alephone/actions/runs/7366266011

Thanks in advance for any help

[dg0yt]

FTR vcpkg CI was also slowed down app. one month ago. But there was no clear cause, because the behaviour wasn't 100% consistent. And the problem disappeared again. The record starts here: #35331 (comment)

[Kolfering]

For information, upgrading the ffmpeg version does not change anything. Tested with 5.1 and 6.1 and it takes the same huge amount of time.

[MonicaLiu0311]

@BillyONeal @JavierMatosD Could you please look at this question?

[laggykiller]

I am encountering same problem, and seems like it only affect manifest mode on GitHub action. Using vcpkg install ffmpeg[all]:x64-windows has normal build time

[laggykiller]

I have done git bisect, and 1181756 is the first bad commit

[dg0yt]

Thanks @laggykiller. You are pointing at the msys update. In this context, I suggest testing with a different version of pkg-config / pkgconf.
All 2.x release of pkgconf are ... troublesome, but is difficult to movet things forward. I could prepare a PR which adds upstream PRs to vcpkg's build of pkgconf, but it would need to be tested someone else than me. In particular pkgconf/pkgconf#339 is just about slow runs in complex situations (ChromeOS) and might very well fit to the ffmpeg port.

[laggykiller]

@dg0yt For testing, do you need help to create a repo with github action that build ffmpeg with different version of vcpkg?

Edit: Here is the repo for quick testing (Feel free to fork it): https://github.com/laggykiller/vcpkg-ffmpeg-build

[laggykiller]

@dg0yt Seems like the cause is msys2-runtime, not pkgconf

This commit of vcpkg-ffmpeg-build uses this commit of vcpkg (Comparison with first bad commit: https://github.com/laggykiller/vcpkg/compare/65fdc55..1181756), in which pkgconf is replaced with a build of the traverse_id pull request you mentioned. The github action for that commit hangs.

On the other hand, This commit of vcpkg-ffmpeg-build uses this commit of vcpkg (Comparison with first bad commit: https://github.com/laggykiller/vcpkg/compare/8cb1c98..1181756), in which only msys2-runtime was downgraded. The github action for that commit does not hang. Hence the culprit is upgraded msys2-runtime package.

[dg0yt]

@laggykiller I already looked at msys2-runtime: #35543. There are some builds which are not affected,

[laggykiller]

@dg0yt The increased build time is quite significant and consistent for ffmpeg in manifest mode based on the >50 builds I had just done. Even if some builds are not affected do not mean it is not an issue, something is wrong with msys2-runtime... Thought of rolling back msys2-runtime?

[dg0yt]

Find the bug in msys2-runtime and fix it. They will eventually remove the old package once an update is available. And there are many vcpkg users which don't update earlly, so vcpkg can't really roll back and wait for the last moment.
Or find the bug in GH Actions.

[laggykiller]

Good news, the latest msys2-runtime 3.4.10-4 seems to have fixed it

https://github.com/laggykiller/vcpkg-ffmpeg-build/actions/runs/7731312275

[dg0yt]

I can prepare an msys2 update in vcpkg. Again, this should be tested in your environment. (I would have waited for a new pkgconf package, but I don't see it coming soon.)

[dg0yt]

I can prepare an msys2 update in vcpkg. Again, this should be tested in your environment.

#36548

[Kolfering]

I tested the PR and can also confirm it's fixed there: https://github.com/Kolfering/alephone/actions/runs/7784312779

[dg0yt]

#36548 was merged.

[MonicaLiu0311]

Fixed by #36548.