[FEA] Improve What Is Stored In CI Artifacts to Reduce Size and Allow for In-Place Builds #112
Assignees
Labels
feature request
New feature or request
Comments
ghost
closed this as
ghost
pushed a commit
that referenced
this issue
Nov 4, 2022
Builds two different images: `morpheus-ci-driver` and `morpheus-ci-test`. The driver image is 15.9GB and the test image is 14.2GB. In this setup only the wheel is transferred from the build to the test and documentation stages(#112). Base image contains: - Pre-populated conda env - libnuma1 Driver image contains: - libnvidia-compute-495 - ibcublas-dev-11-5 - libcufft-dev-11-5 - libcurand-dev-11-5 - libcusolver-dev-11-5 Test image contains - pytest-kafka - Kafka - openjdk-11 (needed for kafka) - nodejs - npm - camouflage fixed #112 fixes #400 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #408
This issue was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
The original implementation of the CI system in PR #80, saves the entire conda environment and workspace as artifacts to allow for the built morpheus packages to be used in docs and testing. It seems that this is necessary because the cudf packages are installed into the conda environment and the morpheus packages are installed as in-place developer builds. This requires both the workspace and conda environment to be zipped up and moved to the next build stage.
Describe the solution you'd like
Taring up the entire conda environment is a bit unnecessary and will require 4+ GB of storage per build when the majority of the files are the same. Instead, we only need a few packages to be transferred between the build and test/docs phase:
cudfconda packagemorpheusconda packagemorpheuswheel packageThe text was updated successfully, but these errors were encountered: