Struggling with simulation setup

[alexpghayes]

Thank you for this wonderful package! I am excited to become more capable with it!

I am working on simulations at https://github.com/alexpghayes/citation-impute-simulations/tree/no_fastadi (should be fully reproducible). So far I have a basic targets workflow running, but I am struggling to generalize it to meet the full needs of my simulation.

My current approach goes like this:

First I take a bunch of model parameters and combine them together into a tibble of model parameters. Then, for each set of model parameters, I sample some data from the model with those parameters. For each dataset sampled from the model associated with a particular set of parameters, I compute three different estimates. Then I compare these estimates to the population values (which also take some computation to compute). Finally I combine all the losses together.

I am struggling with:

1. My model1_parameters data frame has 60 rows and my combined_losses has 180 rows, because there is one loss for each estimator. How do I re-associate each loss with information about the model parameters? (Side note: I realize I could pass the model parameters to the loss function and return them with other loss information, but this seems a bit like a hack)
2. Currently I only sample once for each combination of parameters. In practice I would like replicate the analysis by sampling num_replications times from each set of parameters. That is, I want to programmatically replicate the following section. I came across tar_rep() in tarchetypes but it doesn't seem like it can handle this situation.

[wlandau]

Hi Alex!

(Side note: I realize I could pass the model parameters to the loss function and return them with other loss information, but this seems a bit like a hack)

I actually think this is a perfectly good pattern as long as the parameters can be cleanly joined on and do not take up much space. In fact, I often recommend it in situations that require postprocessing the output and parameters together. Otherwise, tar_branches() may help trace the relationships between the individual branches of dynamically branched targets such as estimate_full_svds and full_sample_graph. Branch provenance is hard when the branching is dynamic.

Currently I only sample once for each combination of parameters. In practice I would like replicate the analysis by sampling num_replications times from each set of parameters. That is, I want to programmatically replicate the following section. I came across tar_rep() in tarchetypes but it doesn't seem like it can handle this situation.

Since you already create a bunch (I assume hundreds) of targets with dynamic branching, it might be appropriate to just perform all reps within each of the existing branches (e.g. replicate(num_replications, run_model(params)) instead of run_model(params)). Defining good targets is a balance between the overhead of many targets and vulnerability to invalidation. So if you do want to use dynamic branching across reps, I recommend batching. https://github.com/wlandau/targets-stan is an example of a workflow custom batching, using targets index_batch and index_sim to define the batching scheme. Your patterns might look something like pattern = map(index_batch, parameter_set).

I am working on simulations at https://github.com/alexpghayes/citation-impute-simulations/tree/no_fastadi (should be fully reproducible).

I cannot access the repo.

[alexpghayes]

I cannot access the repo.

I did everything to make it read for consumption except make the repo public facepalm. Public now.

Since you already create a bunch (I assume hundreds) of targets with dynamic branching, it might be appropriate to just perform all reps within each of the existing branches (e.g. replicate(num_replications, run_model(params)) instead of run_model(params)).

I presume that I would handle the replication myself, essentially by creating and working with lists of results. i.e. targets does not provide a tool to automatically replicate beyond tar_rep()?

[wlandau]

Yup, tar_rep() wraps a very simple kind of batching. Other target factories like tar_stan_mcmc_rep_summary() in stantargets cover more involved situations for specific use cases. All these target factories completely abstract away dynamic branching to avoid confusion for most people.