How to manage repetitive dvc run commands (like unpacking of many zip files)?

[sotte]

Assume you download a bunch of zips.

data/
  download/
    file1.zip
    ...
    fileN.zip

Now you need to unpack the files to lets say data/unpacked/file{i}. Something like this:

data/
  unpacked/
    file1/
    ...
    fileN/

When I use a Makefile to manage my pipeline I would write a simple rule for unpacking the zips and put them into their target folders. What is the best approach to do something like this with DVC? Right now I just type a bunch of dvc run commands but that does not scale well.

[efiop]

Hi @sotte !

Very interesting scenario! We have been thinking about introducing a build matrix #1018 to handle a bit different scenario, but i think in your case it could be useful as well. For example, Dvcfile placed in data/unpacked could look like:

cmd: tar -xf {deps} -C {outs}
matrix:
   - deps:
        - path: file1.tgz
     outs:
        - path: file1
   - deps:
        - path: file2.tgz
     outs:
        - path: file2
    ....

Would something like that suit you? Please feel free to share any thoughts or suggestions.

Thanks,
Ruslan

[sotte]

That looks quite nice!
Wildcards would be nice. Something like this:

matrix:
  - deps:
    -path: file*.csv.tgz
    outs:
    - path: file*.csv

But I guess the deps and outs have to be specified explicitly.

[efiop]

Glad you like it! We will take a closer look at implementing this in the near future.

Yeah, the wildcards raise more questions than answers. I can't promise that they are going to be implemented this way, but we will sure think about it to see if there are some suitable ways we could incorporate them in the future.

Thanks,
Ruslan

[efiop]

Though without wildcards or something similar to them, we still leave the pain point of having to explicitly specify deps and outs. Need to definitely think about optimizing this as well.

[sotte]

True, but if you add the wildcard feature, then people (including myself ;)) would want more data pipeline features. How about rewrite rules? Parallel execution? Feature parity to makefile...at least ;) Do you want to go down the route?

I'm not saying that you should or should not. Just that it's a tricky balance.

[efiop]

Well, we are already sorta going that way with build matrix and upcoming parallel execution, so we might(and I think we should) as well make another step 🙂 Btw, what do you mean by "rewrite rules" ?

[sotte]

Wrong term, sorry. I was thinking of substitutions in makefiles. See https://www.gnu.org/software/make/manual/make.html#Substitution-Refs

[efiop]

Ah, got it. Yes, i think it might be necessary as well in the long term.

[efiop]

Btw, @shcheklein noticed that we had a similar discussion recently https://discuss.dvc.org/t/creating-an-aggregate-dvc-file/93 and thought that it might be worth mentioning that you can workaround this by using a bash script just like in the linked discussion.

[sotte]

That could work. It feels like a dirty workaround though and not very intuitive.

I'll try it out and let you know how it works out in the long run.

[dmpetrov]

Great discussion! The first implementation is coming #4734. It does not completely solve this use case but it is a foundation for the next step - #331.

Let's close this issue and move all the discussions to "umbrella" issues #3633 & #331.