Subprojects?

[krlmlr]

Document best practices to run drake subprojects in the context of a larger drake project. Need to wait for an actual use case.

[wlandau]

The best practices vignette is a good place for our recommendations. The hard part is deciding what subprojects should look like.

[wlandau]

@dapperjapper, do you have more to say on this (from #253)?

[krlmlr]

My current use case: checking downstream dependencies (revdepchecks). I have a static basic plan which I use to construct the dependency graph (as a named list). The results from processing this plan are used to create the actual plan, but I also need intermediate results from the basic plan as arguments to rules in the generated plan. I have found that using readd() in rules doesn't work for jobs > 1 (with "error reading from connection"), now trying to include the basic plan when processing the actual plan.

I'm happy to contribute a vignette that describes the process.

I think we should keep this use case in mind when designing the DSL (#233). Ideally, we'll be able to express the entire process (generating dependencies and constructing the plan from these dependencies) in a single drake_plan() call (or equivalent).

[wlandau]

I see, your plan is changing as it's executing. My current practice in this situation is to chain several make()s together, but this is suboptimal. Right now, I think #233 is the place to work on that aspect of subprojects.

For readd() in workflow plans, are you passing in an existing cache object? Because I think it needs to call get_cache() or something similar, which calls storr::storr_rds() all over again. Maybe I don't totally understand all the details of the use case, but I am also curious as to why you need readd() as opposed to referencing one of the available target names. I would think that loadd() and readd() in commands would spoof the dependency network.

[krlmlr]

I wanted to make the two projects independent of each other, but ultimately decided to include the basic plan into the full plan and avoid readd(). This problem seems to require some more thinking and practice.

[wlandau]

Should we table this issue until we solve #233? Even if we document subprojects now, the best practices will change after #233.

[krlmlr]

Agree to postpone this.

[wlandau]

Sounds good.