How to handle lists in FHIRflat (tracker/discussion)

[pipliggins]

Due to the nested & flexible nature of FHIR, there are several components within multiple Resources which contain lists, or multiple layers of nested lists of data.

The purpose of this issue is to collate and track examples of this behaviour, with the aim of finding a solution to map this data into a flat or semi-flat format while limiting the number of restrictions we have to place on the base FHIR resources.
For example, we could restrict the cardinality of some variables to 0..1 rather than 0..* to remove some of the nested lists, but if we wish to ingest FHIR data in the future this may make it incompatible. We would prefer to simply extend FHIR where necessary, rather than restrict existing functionality.

Part of #1

[pipliggins]

Condition class

BodySite

"bodySite": [
        {
            "coding": [
                {
                    "system": "http://snomed.info/sct",
                    "code": "85151006",
                    "display": "Structure of left hand",
                }
            ],
            "text": "left hand",
        },
        {
            "coding": [
                {
                    "system": "http://snomed.info/sct",
                    "code": "7769000",
                    "display": "Structure of right foot",
                }
            ],
            "text": "right foot",
        }
    ],

[pipliggins]

Encounter class

reason

"reason": [
        {
            "value": [
                {
                    "concept": {
                        "text": "The patient seems to suffer from bilateral pneumonia and renal insufficiency, most likely due to chemotherapy."
                    }
                }
            ]
        }
    ],

diagnosis

"diagnosis": [
        {
            "condition": [{"reference": {"reference": "Condition/stroke"}}],
            "use": [
                {
                    "coding": [
                        {
                            "system": "http://terminology.hl7.org/CodeSystem/diagnosis-role", 
                            "code": "AD",
                            "display": "Admission diagnosis",
                        }
                    ]
                }
            ],
        },
        {
            "condition": [{"reference": {"reference": "Condition/f201"}}],
            "use": [
                {
                    "coding": [
                        {
                            "system": "http://terminology.hl7.org/CodeSystem/diagnosis-role", 
                            "code": "DD",
                            "display": "Discharge diagnosis",
                        }
                    ]
                }
            ],
        },
    ],

[abhidg]

As long as these fields do not need to be searched, it should be fine to store them nested -- for certain fields like bodySite that are 0..* CodeableConcept an additional field bodySite_flat could be created as a list of codes. This way bodySite can be used to go back to FHIR, while bodySite_flat is for efficient searches. I don't think we should create _flat versions of everything as most of these fields will not be used for standard analyses.

[pipliggins]

Observation class

Component

{"component": [
        {
            "code": {
                "coding": [
                    {
                        "system": "http://loinc.org",
                        "code": "8480-6",
                        "display": "Systolic blood pressure",
                    },
                    {
                        "system": "http://snomed.info/sct",
                        "code": "271649006",
                        "display": "Systolic blood pressure",
                    },
                    {
                        "system": "http://acme.org/devices/clinical-codes",
                        "code": "bp-s",
                        "display": "Systolic Blood pressure",
                    },
                ]
            },
            "valueQuantity": {
                "value": 107,
                "unit": "mmHg",
                "system": "http://unitsofmeasure.org",
                "code": "mm[Hg]",
            },
            "interpretation": [
                {
                    "coding": [
                        {
                            "system": "http://terminology.hl7.org/CodeSystem/v3-ObservationInterpretation",
                            "code": "N",
                            "display": "normal",
                        }
                    ],
                    "text": "Normal",
                }
            ],
        },
        {
            "code": {
                "coding": [
                    {
                        "system": "http://loinc.org",
                        "code": "8462-4",
                        "display": "Diastolic blood pressure",
                    }
                ]
            },
            "valueQuantity": {
                "value": 60,
                "unit": "mmHg",
                "system": "http://unitsofmeasure.org",
                "code": "mm[Hg]",
            },
            "interpretation": [
                {
                    "coding": [
                        {
                            "system": "http://terminology.hl7.org/CodeSystem/v3-ObservationInterpretation",
                            "code": "L",
                            "display": "low",
                        }
                    ],
                    "text": "Below low normal",
                }
            ],
        },
    ],
}

[pipliggins]

ResearchStudy resource

outcomeMeasure

"outcomeMeasure": [
        {
            "name": "Single Dose and Food Effect Safety Outcome Measures",
            "type": [
                {
                    "coding": [
                        {
                            "system": "http://terminology.hl7.org/CodeSystem/research-study-objective-type",  # noqa: E501
                            "code": "primary",
                            "display": "Primary",
                        }
                    ]
                }
            ],
            "description": "Incidence and severity of adverse events (AEs), laboratory, ECG, and vital sign changes",  # noqa: E501
            "reference": {
                "type": "EvidenceVariable",
                "identifier": {
                    "type": {"text": "FEvIR Linking Identifier"},
                    "system": "https://fevir.net/FLI",
                    "value": "NCT05503693-primaryOutcome-0",
                    "assigner": {"display": "Computable Publishing LLC"},
                },
            },
        },
        {
            "name": "Multiple Dose Safety Outcome Measures",
            "type": [
                {
                    "coding": [
                        {
                            "system": "http://terminology.hl7.org/CodeSystem/research-study-objective-type",  # noqa: E501
                            "code": "primary",
                            "display": "Primary",
                        }
                    ]
                }
            ],
            "description": "Incidence and severity of AEs, laboratory, ECG, and vital sign changes.",  # noqa: E501
            "reference": {
                "type": "EvidenceVariable",
                "identifier": {
                    "type": {"text": "FEvIR Linking Identifier"},
                    "system": "https://fevir.net/FLI",
                    "value": "NCT05503693-primaryOutcome-1",
                    "assigner": {"display": "Computable Publishing LLC"},
                },
            },
        },
        {
            "name": "Cmax after single dose",
            "type": [
                {
                    "coding": [
                        {
                            "system": "http://terminology.hl7.org/CodeSystem/research-study-objective-type",  # noqa: E501
                            "code": "primary",
                            "display": "Primary",
                        }
                    ]
                }
            ],
            "description": "PK characteristics after single dose",
            "reference": {
                "type": "EvidenceVariable",
                "identifier": {
                    "type": {"text": "FEvIR Linking Identifier"},
                    "system": "https://fevir.net/FLI",
                    "value": "NCT05503693-primaryOutcome-2",
                    "assigner": {"display": "Computable Publishing LLC"},
                },
            },
        },
...

progressStatus

"progressStatus": [
        {
            "state": {
                "coding": [
                    {
                        "system": "http://hl7.org/fhir/research-study-status",
                        "code": "recruiting",
                        "display": "Recruiting",
                    }
                ]
            }
        },
        {
            "state": {
                "coding": [
                    {
                        "system": "http://hl7.org/fhir/research-study-status",
                        "code": "overall-study",
                        "display": "Overall Study",
                    }
                ]
            },
            "actual": False,
            "period": {"start": "2022-12-06", "end": "2023-06"},
        },
    ],

comparisonGroup

"comparisonGroup": [
        {
            "linkId": "ap303",
            "name": "AP303",
            "type": {"text": "Experimental"},
            "description": "AP303",
            "intendedExposure": [
                {
                    "reference": "#NCT05503693-drug------ap303-50-g",
                    "type": "EvidenceVariable",
                    "display": "Drug: AP303 50 μg",
                },
                {
                    "reference": "#NCT05503693-drug------ap303-150-g",
                    "type": "EvidenceVariable",
                    "display": "Drug: AP303 150 μg",
                },
                {
                    "reference": "#NCT05503693-drug------ap303-300-g",
                    "type": "EvidenceVariable",
                    "display": "Drug: AP303 300 μg",
                },
                {
                    "reference": "#NCT05503693-drug------ap303-600-g",
                    "type": "EvidenceVariable",
                    "display": "Drug: AP303 600 μg",
                },
            ],
        },
        {
            "linkId": "placebo",
            "name": "Placebo",
            "type": {"text": "Placebo Comparator"},
            "description": "Placebo",
            "intendedExposure": [
                {
                    "reference": "#NCT05503693-drug------placebo-50-g",
                    "type": "EvidenceVariable",
                    "display": "Drug: Placebo 50 μg",
                },
                {
                    "reference": "#NCT05503693-drug------placebo-150-g",
                    "type": "EvidenceVariable",
                    "display": "Drug: Placebo 150 μg",
                },
                {
                    "reference": "#NCT05503693-drug------placebo-300-g",
                    "type": "EvidenceVariable",
                    "display": "Drug: Placebo 300 μg",
                },
                {
                    "reference": "#NCT05503693-drug------placebo-600-g",
                    "type": "EvidenceVariable",
                    "display": "Drug: Placebo 600 μg",
                },
            ],
        },
    ],

[pipliggins]

Currently resolved by storing fields with lists containing more than one data item as nested data items, with a "_dense" suffix, as these fields are not expected to be queried regularly in the standard analyses. If codeableConcept fields with 0..* cardinality are found to occur regularly in the input data & in searches, another column containing only the codes in url|code format could be added in a separate pull request.