-
-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extremely high CPU use by ffmpeg #75
Comments
Notes:
Something simpler may help. This targets a bitrate of 300kb within a 512kb window and gives a wider quality range: At least for me, the second one is generates faster, looks better, and consumes about 1/3 the bandwidth. There's no minimum bitrate and no maximum q so it can fly past all of those motionless whiteboard images.
|
Thank you for reporting this and doing some tests. I will have a look into it. |
Regarding versions, image
|
Regarding ffmpeg settings, @rgaudin @kelson42 do we have any past issue which discuss why these settings have been chosen? I imagine finding one preset to more or less rule them all is not an easy feat. Encoding logic is coming from python-scraperlib and presets (we use the low quality webm version for Khan Academy recipe) Webm low quality presets are coming from openzim/python-scraperlib#14 (openzim/python-scraperlib@78e2bb0#diff-2cc68edde814805fe24114313acdde91ae832adef02f7d0576675d74db3f7b58 more precisely) but I did not find any discussion there, so they probably have been ported from ted/youtube scrapers, but I failed to find any discussion over there. |
|
I tested suggested settings on https://studio.learningequality.org/content/storage/b/7/b71ca7f102ae16e4023c9f49b015d6b7.mp4 I do not find a significant visual difference in the resulting file (but this is obviously very personal). I confirm that processing is a little faster (from 10secs to 8secs) and file is more than 3 times smaller (from 2.7MB to 768KB, while original mp4 is 690KB). I do not find any difference (in terms of processing time) between in Docker and on the host directly (same machine), so there is probably something strange/unusual in your Docker setup on your machine. |
vp9 or vp8? looking at the setting I believe we use vp8 |
Sorry it's a slip, vp8 of course |
Progressing towards a merged PR on this will obviously needs significant testing with many kind of videos and we (@kiwix) probably won't have sufficient bandwidth for this in the coming months. Contributions are of course more than welcomed. Note however that this effort might conflict with another initiative we might consider to start around choosing a different video codec (and JS libs to fallback when reader/browser does not support this codec). The test set (and testing procedure) will nevertheless be very useful and most probably reused. |
I was going to kill this task on pixelmemory because it has built up over 161 GB of files...but not really. The filesystem compression ratio is over 3:1 so it's only 52GB on disk. That should not be happening for video files. |
At this stage it looks like we might move from webm/vp8 to mpg4/h264. If we go that direction, we should reassess our ffmpeg command line (in particular for low quality). |
I can only agree. And it match the 3:1 ratio we both observed when changing the ffmpeg settings. I compressed (with default Zip settings on Mac) the "big" video I previously obtained with current scraper ffmpeg settings and I confirm it compress very well (again a 3:1 ratio, going from 2.7M to 868MB) which shouldn't be possible for a video file. |
Edit: 868KB, not 868MB |
MPEG4 to WebM transcoding is extremely slow. https://farm.openzim.org/pipeline/c06a8148-7d9a-422c-b5b4-abfe93d51168 has been crawling along for two weeks while using 100% of all CPUs.
The text was updated successfully, but these errors were encountered: