Feature/sda download #392
Merged
Feature/sda download #392
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Download a file and check the path and the file itself - Remove s3cmd-admin because it is the same with the directS3 - Renamed the s3cmd.conf to template and then different tests can create a config and add different tokens - Fixed the url check and fail if there is not protocol (https or http)
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #392 +/- ##
==========================================
+ Coverage 40.48% 42.25% +1.77%
==========================================
Files 11 12 +1
Lines 1539 1678 +139
==========================================
+ Hits 623 709 +86
- Misses 837 878 +41
- Partials 79 91 +12
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
kostas-kou
requested review from
MalinAhlberg,
nanjiangshu,
pahatz,
jbygdell and
aaperis
jbygdell
requested changes
May 30, 2024
- the cli now checks the format of the first part of the filepath and if it is a userid like does not include it - add a userid in the filepath in integration setup
jbygdell
reviewed
Jun 3, 2024
Co-authored-by: Joakim Bygdell <joakim.bygdell@nbis.se>
jbygdell
reviewed
Jun 4, 2024
MalinAhlberg
previously approved these changes
Jun 11, 2024
jbygdell
previously approved these changes
Jun 14, 2024
Co-authored-by: Malin Klang <ahlberg.malin@gmail.com>
MalinAhlberg
approved these changes
Jun 14, 2024
aaperis
approved these changes
Jun 14, 2024
nanjiangshu
reviewed
Jun 14, 2024
nanjiangshu
reviewed
Jun 14, 2024
nanjiangshu
reviewed
Jun 14, 2024
nanjiangshu
reviewed
Jun 14, 2024
There was a problem hiding this comment.
It works very well when I tested. A general feeling is that the naming of sub commands download and sda-download does not clearly distinguish methods for downloading.
I don't have any good suggestions neither. Maybe
s3-download: by downloading from a S3 bucket
api-download: by using the download API
Co-authored-by: Alex Aperis <76202622+aaperis@users.noreply.github.com>
Update the download environment variables for working with the latest image
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related issue(s) and PR(s)
This PR closes #372.
Description
A new feature in the sda-cli is implemented to download files from the download service endpoint by giving the dataset stable_id and the inbox path of the file without the uploader's user id
How to test
By testing that just run the setup script from the root folder:
bash .github/integration/setup/setup.shand when it is done one can download the file by running:
./sda-cli sda-download -config testing/s3cmd-download.conf -dataset https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir test-download main/subfolder/dummy_data.c4ghThen check if the file
test-download/main/subfolder/dummy_dataexists and if the result by runninghead test-download/main/subfolder/dummy_datais "THIS FILE IS JUST DUMMY DATA"