Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement SIF ORAS protocol to HTTP #5000

Open
wants to merge 58 commits into
base: master
Choose a base branch
from
Open

Implement SIF ORAS protocol to HTTP #5000

wants to merge 58 commits into from

Conversation

pditommaso
Copy link
Member

This PR draft the mapping of ORAS images names to HTTP ones.

@pditommaso pditommaso marked this pull request as draft May 15, 2024 20:10
@netlify Netlify
Copy link

netlify bot commented May 15, 2024 •
edited
Loading

✅ Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit a46d4a1
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/667051543a4e0900082c777a
😎 Deploy Preview https://deploy-preview-5000--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@pditommaso pditommaso mentioned this pull request May 15, 2024
Open
pditommaso and others added 27 commits May 31, 2024 14:27
@pditommaso @marcodelapierre
Use protected visibility for updateStatus method
01aed69 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Bump nf-amazon@2.5.1
9a6d1ea 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Fix unexpected container resolution
1fa3f0a 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Fix inspect should not write history entry
051f59f 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Implement SIF ORAS protocol to HTTP
d9167e6 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
wip
a7a01a8 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Fix Taskbar API is not supported error [ci fast]
e7512b0 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Improve icon loading error handling [ci skip]
c9f2e8d 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@greenberga @pditommaso @marcodelapierre
Add 'preview' to workflow runtime metadata (#4985)
0d3a8b1 
Signed-off-by: Aaron Greenberg <p@aaronjgreenberg.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@bentsherman @pditommaso @marcodelapierre
Workflow output definition (#4784)
f7b4cc0 
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Remove seqera and defaults from Conda default channels (#5003)
0cd9c1c 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Bump Fusion 2.3 (#5005)
7e271fc 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@bentsherman @pditommaso @marcodelapierre
Remove Fusion symlink resolution (#5004)
0bbf1e9 
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Bump groovy-console 4.0.21-patch.2 [ci fast]
ad1f9f2 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Add enabled property to output dsl (#5008) [ci fast]
4804eb9 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Bump nf-console@1.1.3
ce18c4a 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Bump nf-google@1.13.1
e364af1 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Bump nf-wave@1.4.2
5114889 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Update changelog
a543cf8 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
[release 24.04.0] Update timestamp and build number [ci fast]
6a6a818 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Fix nf-amazon plugin dependecy
deb9315 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Fix nf-google plugin dependecy
4749fa5 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Bump nf-amazon@2.5.2
ce29008 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Bump nf-google@1.13.2
6ba8f47 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
[release 24.04.1] Update timestamp and build number [ci fast]
e9ad651 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Fix typos [ci skip]
9a05175 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
WaveClient: fix using registerTypeAdapter for Instant class
7cf5bd1 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
pditommaso and others added 12 commits May 31, 2024 14:27
@pditommaso @marcodelapierre
Fix handling of tags with slash on Bitbucket (#5031) [ci fast]
0a8ebbd 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Update changelog
3bd4fd9 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
[release 24.04.2] Update timestamp and build number [ci fast]
1dbe8e7 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Fix Java 22 version in launcher [ci skip]
42ab4ee 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
updated some unit tests
1a989e2 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
update WaveClientTest
b4f6b18 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Revert "update WaveClientTest"
e833800 
This reverts commit 947908e.

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
draft alternate implementation - to be tested
e6b16cb 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Http vs Oras, impl foucsed on ContainerInspectMode, with unit tests
8b3552b 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
couple of fixes
7f78d1a 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
fix for CmdInspectTest vs ContainerHandlerTest
c4db13a 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso @marcodelapierre
Enable build with Java 21 (#5030)
4d1bf0e 
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Merge branch 'master' into wave-sif-http
b7ac53a 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Copy link
Member

( a lot of extra commits due to fixing commit sign-offs - sorry! )

@marcodelapierre marcodelapierre marked this pull request as ready for review May 31, 2024 06:29
@marcodelapierre
Copy link
Member

marcodelapierre commented May 31, 2024 •
edited
Loading

@pditommaso ready for review

  • Correctly shows HTTP vs ORAS URIs
  • Warning when requesting HTTP and image not found
  • Concretize now actually concretises Wave image builds
  • Unit tests updated

Maybe we can still improve on messages/behaviour when image is not built, but otherwise the key implementation changes are there.

@marcodelapierre
Copy link
Member

marcodelapierre commented Jun 3, 2024 •
edited
Loading

Still one failing unit test on CI should resolve docker container with wave container in WaveContainerResolverTest, call to resolveImage at line 76.

The responsible bit of code is in WaveContainerResolver, new lines 98-108.

Looking into it.

@marcodelapierre
modules/nextflow/src/test/groovy/nextflow/container/ContainerConfigTe…
8a2756b 
…st: test more verbose

Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Merge branch 'master' into wave-sif-http
eca8e2f 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
WaveClient singularityOrasToHttp: workaround for Seqera Containers
2d8d356 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Copy link
Member

@pditommaso @ewels following the meeting today, I have implemented the workaround for the URI in the case of Seqera Containers: https://github.com/nextflow-io/nextflow/blob/wave-sif-http/plugins/nf-wave/src/main/io/seqera/wave/plugin/WaveClient.groovy#L392-L400

Ready for review (with one last unit test to fix)

@marcodelapierre
fixed last unit test
95ed7f9 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Merge branch 'master' into wave-sif-http
a168372 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@marcodelapierre
Copy link
Member

OK, last outstanding unit test fixed:

  • local tests OK
  • github CI running now.

@marcodelapierre
Copy link
Member

@pditommaso ready for review

@marcodelapierre
Copy link
Member

Do we need an integration test for running with Wave plugin and using Http over Oras?

@pditommaso
Copy link
Member Author

It may be useful. Some wave tests are here

@marcodelapierre
Merge branch 'master' into wave-sif-http
a46d4a1 
Signed-off-by: Dr Marco Claudio De La Pierre <marco.delapierre@gmail.com>
@pditommaso
Copy link
Member Author

Let's put in hold, not sure anymore it makes sense to implement this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@pditommaso @marcodelapierre @greenberga @bentsherman @nschan