Data Specifications in Workflows

[jwildfire]

Overview

As we start extending {gsm} with new apps and modules, capturing some level of data requirements is needed - especially for domain-level mapping data. I think there are a few ways we could do this:

1. Extend workflows using a new dataSpec property or something similar
2. Use a different validation framework - maybe json schema or something from arrow/parquet ...
3. Use an existing R package
   • assertr
   • data.validator
   • validate
   • pointblack

To extend the yaml, I think capturing required tables and required/optional columns is probably sufficient. To start very simple, we could add the following to kri0001.yaml (the AE KRI) to indicate the required columns:

spec: 
  dfAE:
    - subjid
  dfEnrolled:
    - subjid
    - invid
    - timeonstudy  

Tracking required/optional columns would be slightly more complex. We could just add a string (or boolean) to each column, or we could make a more complex object.

# With string
spec: 
  dfAE:
    subjid: required
  dfEnrolled:
    subjid: required
    invid: optional
    timeonstudy: optional

# With Boolean
spec: 
  dfAE:
    subjid: TRUE
  dfEnrolled:
    subjid: TRUE
    invid: FALSE
    timeonstudy: FALSE

# with a required property (Also allows typing, etc
spec: 
  dfAE:
    subjid:
       required: TRUE
       type: string
...

Leaning towards option #1 as a baseline implementation. Will prototype something ASAP.

[jwildfire]

I'm thinking of adding data domain, to the data.frame names in our workflows. A few examples:

• In inst/workflows/data_mapping.yaml, dfAE becomes Raw_AE
• In inst/workflows/metrics.yaml, dfSeriousAE becomes Mapped_SeriousAE and dfAE becomes Mapped_AE
• In inst/workflows/reports.yaml, dfResultsbecomesReporting_Results`

This allows a lot of additional transparency about exactly with part of the data model is being queries, and would also allow for nice composite specs that pull in requirements across all workflows. Something like:

combine_specs <- function(lSpecs){
    all_specs <- list()
    for (spec in lSpecs){
        for (domain in names(spec)){
            if (domain %in% names(all_specs)){
                all_specs[[domain]] <- c(all_specs[[domain]], spec[[domain]])
            } else {
                all_specs[[domain]] <- spec[[domain]]
            }
        }
    }
    return(lSpec)
}

all_wf <- gsm::MakeWorkflowList()
all_specs <- combineSpecs(all_wf)

Without the refactor, it's going to be hard to tell where data.frames in names(all_specs) fall in the data model since everything is prefixed with df. For example, there would be no way to know if dfAE comes from Raw or Mapped (In fact, currently it's in both). With the naming update, names would all be prefixed with raw_, mapped_, analysis_ or results_ which is nice and clean.

[jwildfire]

cc: @lauramaxwell @jonthegeek @samussiah. I don't think this is a breaking change (so still ok for v2.1), but would probably break some tests and require some doc updates ...

[jwildfire]

cc: @dpastoor as well

[jwildfire]

Basic prototype in place in #1746

[jwildfire]

Next Steps:

• Add unit tests for CombineSpecs and CheckSpec
• Update CheckSpec to take required parameter into account
• Update specs to add type parameter and incorporate in CheckSpec
• Refactor mapping.yaml to support new spec framework.