forked from apache/airflow
-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Switches to "/" convention in ghcr.io images with optimisations
We are using ghcr.io as image cache for our CI builds and Breeze and it seems ghcr.io is being "rebuilt" while running. We had been using "airflow-<branch>.." image convention before, bacause multiple nesting levels of images were not supported, however we experienced errors recently with pushing 2.1 images (https://issues.apache.org/jira/browse/INFRA-22124) and during investigation it turned out, that it is possible now to use "/" in the name of the image, and while it still does not introduce multiple nesting levels and folder structure, the UI of GitHub treats it like that and if you have image which starts wiht "airflow/", the airflow prefix is stripped out and you can also have even more "/" in then name to introduce further hierarchy. Since we have to change image naming convention due to (still unresolved) bug with no permission to push the v2-1-test image we've decided to change naming convention for all our cache images to follow this - now available - "/" connvention to make it better structured and easier to manage/understand. Some more optimisations are implemented - Python, prod-build and ci-manifest images are only pushed when "latest" image is prepared. They are not needed for the COMMIT builds because we only need final images for those builds. This simplified the code quite a bit. CONTINUE_ON_PIP_CHECK_ERROR variable has been removed in favour of ignoring pip error when installing dependencies from branch tip. This might potentially happen for a short while when new changes have been merged, but constraints were not yet regenerated and we have conflicting dependencies. The .dockerignore was reviewed and builds were optimized for people who locally built provider packages and documentation, by excluding unnecessary files. Some instructions which run after the COPY . which did not need sourcer were moved before the COPY command. Those optimisatiions save 30-40 seconds of overhead when building the image (especially when you build images incrementally rather than rebuilding from scratch). PIP and HELM versions have been updated to latest available. Backwards-compatibility was implemented to allow PRs that have not been upgraded to continue building after this one is merged, also a workaround has been implemented to make this change to work even if it is not merged yet to main. This "legacy" mode will be removed in ~week when everybody rebase on top of main. Documentation is updated reflecting those changes. (cherry picked from commit e04c2e3872aa30ed042d3f9bf66d8020cf9c2acb)
- Loading branch information
Showing
33 changed files
with
463 additions
and
422 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.