Heavy screenshot usage in repository (size and number of files)

[MikeMcC399]

Where to find the issue

https://github.com/corona-warn-app/cwa-website/tree/master/src/assets/screenshots

Problem description

The screenshots directory in the cwa-website repository uses a large amount of space and includes a high number of files.

Each app version release causes a complete set of images to be committed to the repository for the combinations iOS/English, Android/English, iOS/German and Android/German. Many of these files have not changed between versions, meaning that there is a high level of duplication.

Metric	Screenshots	Total web
Size	966 MB / 70%	1.38 GB / 100%
Files	11.296 / 88%	12.881 / 100%

Summary

Screenshots cause ...

• High use of space in total
• High number of files in total
• High number of files in each release PR
  • difficult to review
  • GitHub "Files changed" function in PRs does not work well with so many image files

Suggested change

De-duplicate the screenshot images and use a method where only changed screenshots need to be published.

Advantages

• Reduce the size and complexity of the repository
• If only changes need to be published with each release then reviewing the changes will be simpler. Currently more than 600 files are part of each related PR.
• GitHub "Files changed" for reviewing PRs will work better

Internal Tracking ID: EXPOSUREAPP-14534

[MikeMcC399]

cwa-website now takes up a massive 5.3 GB space on disk.

git is not really suited to managing large numbers of image files as it tries to keep differences on each of them. The hidden .git folder is taking up more than 3GB. For comparison the same folder for the Android app is 590MB.

I cannot copy the website over the network any more. I get an error:

error: RPC failed; curl 56 Recv failure: Connection was resets
send-pack: unexpected disconnect while reading sideband packet

I created a copy in order to test workflows separately, so I may not be able to contribute in this area any more. 🙁

It looks like some heavy git maintenance may be necessary!

Edit: Today (Jan 13, 2023) I was able to copy the website. It took quite a long time and I received a warning about large files.

• See issue Large file warning uploading copy of cwa-website to GitHub #3325.

[dsarkar]

@MikeMcC399 ok, we will check, how to remove redundant image files.

[dsarkar]

Internal Tracking ID: EXPOSUREAPP-14534

[MikeMcC399]

@dsarkar

ok, we will check, how to remove redundant image files.

Thank you! Unfortunately I don't think it is quite that simple, because even if files are removed they are still present in the history. To solve the issue it probably means removing files and rewriting the git history. I don't know if that is feasible without breaking things.

[Ein-Tim]

I'd also very much appreciate it if the file size of this repository would be decreased, as it takes 6 GB of disk space on my disk!

[MikeMcC399]

Some things which could be considered:

• remove old versions of screenshots (currently version 2.2 to 2.28 i.e. 27 sets of English/German Android/iOS screenshots)

See git-filter-repo, which has functionality for removing objects from the git history. It also includes the command

git filter-repo --analyze

to investigate the .git space usage. This shows active and deleted file usage. It already shows high space usage of deleted screenshots files. (Analysis report: filter-repo.zip.)

See GitHub help About large files on GitHub including:

• About Git Large File Storage

and references to removing files.

---

I suspect however, that this is not solvable with consideration to a reasonable amount of effort within the lifetime of this project.

---

https://github.com/git-guides#what-is-git

"Git stores changes in SHA hashes, which work by compressing text files. That makes Git a very good version control system (VCS) for software programming, but not so good for binary files like images or videos."

[MikeMcC399]

• I noticed PR Experimental: Test cwa-website symlink handling #3327 related to this issue
• I ask the question: Is it necessary to store a complete history of all screenshot versions? I assume that most users would only be interested in the current version and the previous version of the app.

[larswmh]

@MikeMcC399

I ask the question: Is it necessary to store a complete history of all screenshot versions? I assume that most users would only be interested in the current version and the previous version of the app.

Keeping screenshots for all versions of the app was specifically requested by the stakeholders

[MikeMcC399]

@larswmh

Keeping screenshots for all versions of the app was specifically requested by the stakeholders

Perhaps the stakeholders could reconsider whether they want to continue holding on to this requirement, given the major impact on the GitHub repository? If the stakeholders want the screenshots to be available for themselves, rather than for app users, then it might be possible to set up an alternate archive site for stakeholders use. Then the older versions could be removed from the main site.

[larswmh]

@Ein-Tim @MikeMcC399

It was internally decided that we are not going to put further effort into this issue considering this projects remaining timespan and the unforseeable impact this could have in the long run. Git LFS would not reduce the repository size, but store large files in a more favorable environment for git. Symlinks are a nice workaround but are tedious to work with because of their complexity.

However, as seen for the upcoming version 3.1, we started linking to screenshots of previous versions. In the future, the screenshot size will only grow marginally.

We also decided to keep the git history as it is for transparency reasons.

Thanks for your understanding.

[MikeMcC399]

@larswmh

Many thanks for reviewing the issue and coming to a decision. After researching the issue myself and the impact of various strategies I fully understand the decision not to change very much considering the remaining project time.

It's good that there is now visibility to this issue and that the screenshot linkage will help prevent the problem getting worse.