Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OCI Auto mode in Singularity/Apptainer #4548

Merged
merged 11 commits into from
Dec 17, 2023
Merged

OCI Auto mode in Singularity/Apptainer #4548

merged 11 commits into from
Dec 17, 2023

Conversation

pditommaso
Copy link
Member

Add the ability to run OCI containers with Apptainer without the need to convert them explicitly via ApptainerCache

@pditommaso
Add OCI-mode like for Apptainer
e9aa8b2 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@netlify Netlify
Copy link

netlify bot commented Dec 1, 2023 •
edited
Loading

✅ Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit a5b7aff
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/657f4037f3621600089f7761
😎 Deploy Preview https://deploy-preview-4548--nextflow-docs-staging.netlify.app/config
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@pditommaso pditommaso mentioned this pull request Dec 1, 2023
Closed
@marcodelapierre marcodelapierre self-requested a review December 1, 2023 04:51
@marcodelapierre marcodelapierre self-assigned this Dec 1, 2023
@marcodelapierre
Copy link
Contributor

Hold on, see my thread on the originating issue: #4543

@pditommaso
Merge branch 'master' into apptainer-oci-mode
57e8fed
@pditommaso
Add OCI direct mode
8a92e18 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Fix tests
2193a45 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Copy link
Member Author

To summarise, currently this PR allows the use of OCI (Docker) containers converting implicitly to SIF image file format both for Singularity and Apptainer by adding respectively the option

singularity.direct = true
apptainer.direct = true

Along with this is also supported the native OCI support in singularity 4 by using

singularity.oci = true

However, considering they are both OCI containers may a better naming could be

apptainer.autoOciMode = true|false
singularity.autoOciMode = true|false

and

singularity.nativeOciMode = true|false

OR

apptainer.ociAuto = true|false
singularity.ociAuto = true|false

and

singularity.ociNative = true|false

@marcodelapierre
Copy link
Contributor

Yep, second 100% your summary around the proposed functionalities.
Personally, I like ociAuto and ociNative, as they are concise.
Also, not that ociNative in Singularity implies ociAuto.

I will review this PR on Wed at the latest.

@marcodelapierre marcodelapierre mentioned this pull request Dec 13, 2023
Open
marcodelapierre
marcodelapierre suggested changes Dec 13, 2023
View reviewed changes
Copy link
Contributor

@marcodelapierre marcodelapierre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small requests along the code - thanks Paolo!

modules/nextflow/src/main/groovy/nextflow/container/ContainerConfig.groovy Outdated Show resolved Hide resolved
modules/nextflow/src/main/groovy/nextflow/container/ContainerConfig.groovy Outdated Show resolved Hide resolved
modules/nextflow/src/main/groovy/nextflow/container/ContainerHandler.groovy Show resolved Hide resolved
plugins/nf-wave/src/main/io/seqera/wave/plugin/resolver/WaveContainerResolver.groovy
@@ -70,8 +70,7 @@ class WaveContainerResolver implements ContainerResolver {
final freeze = client().config().freezeMode()
final config = task.getContainerConfig()
final engine = getContainerEngine0(config)
final singularityOciMode = config.singularityOciMode()
final singularitySpec = freeze && engine in SINGULARITY_LIKE && !singularityOciMode
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is SingularitySpec used for in Wave?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see that SingularitySpec controls whether a Singularityfile or Dockerfile is used.
At this stage it is not clear to me why this is conditional to canRunOciImage being false; though this is not in scope for this PR.

@marcodelapierre
Copy link
Contributor

Forgot to mention: these updates to Oci options need to be reflected in the docs as well

@marcodelapierre marcodelapierre changed the title Add OCI-mode like for Apptainer OCI Auto mode in Singularity/Apptainer Dec 14, 2023
jordeu and others added 6 commits December 17, 2023 18:09
@jordeu @pditommaso
Add AWS_SESSION_TOKEN to Fusion environment (#4581)
0003b39 
Signed-off-by: Jordi Deu-Pons <jordi@jordeu.net>
@pditommaso
Fix failing test
4e5bc7c 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Move build num & timestamp to BuildInfo class
03537b2 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@bentsherman @pditommaso
Build optimizations (#4579)
4bd660b 

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman @pditommaso
Fix bug with Fusion symlink resolution (#4593)
9a58236 
This commit is a supplement to PR #4348
to fix the issue #4309

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Improve tests [ci fast]
a5b7aff 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Copy link
Member Author

Ok, considering the term OCI native is used is several places with different meaning, I went for ociMode (that's how it's referred in the docs) and ociAutoPull.

The docs has been updated correspondingly.

@pditommaso pditommaso merged commit b7f1a19 into master Dec 17, 2023
22 checks passed
@pditommaso pditommaso deleted the apptainer-oci-mode branch December 17, 2023 20:12
@marcodelapierre marcodelapierre linked an issue Jan 18, 2024 that may be closed by this pull request
Support OCI images in apptainer #4543
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marcodelapierre marcodelapierre marcodelapierre requested changes

Assignees

@marcodelapierre marcodelapierre

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support OCI images in apptainer
4 participants
@pditommaso @marcodelapierre @jordeu @bentsherman