Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli: support dir and glob pattern on download #490

Merged
merged 2 commits into from
Apr 20, 2021

Conversation

mvidalgarcia
Copy link
Contributor

@mvidalgarcia mvidalgarcia commented Mar 26, 2021

Depends on #483

Needs r-commons release to pass - reanahub/reana-commons#253

closes #402

To test

  • Create a workflow
  • Upload multiple files in multiple levels of directories
  • Try to download different directories by name
  • Try to download multiple files by passing glob patterns

Pseudo-example

$ reana-client download -w workflow 'mydir'
$ reana-client download -w workflow '**/*.py'
$ reana-client download -w workflow '**/*.txt'
$ reana-client download -w workflow 'mydir/foo/*.py'
...

If it the pattern matches more than one file, you should get a zip downloaded, otherwise a single file.

TBD

When downloading a single file, until now, the path where this file is in the workspace is respected locally. This is because the file is saved directly using the value that is passed to the command as an argument. E.g. reana-client download -w workflow foo/bar/myfile.txt would create directories foo/bar for you as it uses this argument to save the file.

After this PR, since the argument can be a glob pattern, we get the filename from the server-side and save it as it is, thus, causing the path where the file is not to be respected. This is done this way because otherwise, one might end up having folders called \*\* or similar. E.g. reana-client download -w workflow '**/myfile.txt' would create a dir called \*\* and put myfile.txt inside.

Update 19/04/2021: It is possible to pass attachment_filename arg to send_from_directory so one can customize the content of filename in the Content-Disposition header. Passing there a relative path allows us to keep the folder structure and be backwards compatible. Updated on https://github.com/reanahub/reana-workflow-controller/pull/369/files

@mvidalgarcia mvidalgarcia changed the base branch from master to maint-0.7 March 26, 2021 12:41
Copy link
Member

@audrium audrium left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, tested it carefully and haven't spotted any issues 🚀

reana ❯ reana-client ls
NAME                                         SIZE     LAST-MODIFIED
fitdata/plot.png                             15450    2021-04-19T14:27:23
_yadage/yadage_snapshot_workflow.json        13274    2021-04-19T14:27:37
_yadage/yadage_workflow_instance.png         33777    2021-04-19T14:27:37
_yadage/yadage_workflow_instance.pdf         13928    2021-04-19T14:27:37
_yadage/yadage_workflow_instance.dot         1797     2021-04-19T14:27:37
code/gendata.C                               1937     2021-04-19T14:27:08
code/fitdata.C                               1648     2021-04-19T14:27:08
code/test.txt                                0        2021-04-19T14:32:03
gendata/data.root                            154859   2021-04-19T14:27:17
_yadage/adage/adagesnap.txt                  1337     2021-04-19T14:27:37
_yadage/adage/workflow.gif                   221433   2021-04-19T14:27:37
workflow/yadage/workflow-htcondorcern.yaml   2025     2021-04-19T14:27:08
workflow/yadage/workflow-slurmcern.yaml      1929     2021-04-19T14:27:08
workflow/yadage/workflow.yaml                1805     2021-04-19T14:27:08

~/Documents/projects/test
reana ❯ reana-client download 'code/*.C'
File download_2021-04-20-081952.zip downloaded to /home/amecioni/Documents/projects/test.

~/Documents/projects/test
reana ❯ unzip download_2021-04-20-081952.zip
Archive:  download_2021-04-20-081952.zip
  inflating: code/gendata.C
  inflating: code/fitdata.C

~/Documents/projects/test
reana ❯ l
drwxrwxr-x    - amecioni 2021-04-20 10:20  code
.rw-rw-r-- 1.8k amecioni 2021-04-20 10:19  download_2021-04-20-081952.zip

~/Documents/projects/test
reana ❯ l code
.rw-rw-r-- 1.6k amecioni 2021-04-19 14:27  fitdata.C
.rw-rw-r-- 1.9k amecioni 2021-04-19 14:27  gendata.C

@mvidalgarcia mvidalgarcia force-pushed the rc/402-dir-download branch 5 times, most recently from 3a7bb1e to 520607d Compare April 20, 2021 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

download: add support for downloading directories
2 participants