03-modality-agnostic-files.md

# Modality agnostic files

## Dataset description

Templates:

-   `dataset_description.json`
-   `README`
-   `CHANGES`
-   `LICENSE`

### `dataset_description.json`

The file `dataset_description.json` is a JSON file describing the dataset.
Every dataset MUST include this file with the following fields:

{{ MACROS___make_metadata_table(
   {
      "Name": "REQUIRED",
      "BIDSVersion": "REQUIRED",
      "HEDVersion": "RECOMMENDED",
      "DatasetType": "RECOMMENDED",
      "License": "RECOMMENDED",
      "Authors": "OPTIONAL",
      "Acknowledgements": "OPTIONAL",
      "HowToAcknowledge": "OPTIONAL",
      "Funding": "OPTIONAL",
      "EthicsApprovals": "OPTIONAL",
      "ReferencesAndLinks": "OPTIONAL",
      "DatasetDOI": "OPTIONAL",
      "GeneratedBy": "RECOMMENDED",
      "SourceDatasets": "RECOMMENDED",
   }
) }}

Each object in the `GeneratedBy` array includes the following REQUIRED, RECOMMENDED
and OPTIONAL keys:

| **Key name** | **Requirement level** | **Data type** | **Description**                                                                                                                                                                                        |
|--------------|-----------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Name         | REQUIRED              | [string][]    | Name of the pipeline or process that generated the outputs. Use `"Manual"` to indicate the derivatives were generated by hand, or adjusted manually after an initial run of an automated pipeline.     |
| Version      | RECOMMENDED           | [string][]    | Version of the pipeline.                                                                                                                                                                               |
| Description  | OPTIONAL              | [string][]    | Plain-text description of the pipeline or process that generated the outputs. RECOMMENDED if `Name` is `"Manual"`.                                                                                     |
| CodeURL      | OPTIONAL              | [string][]    | URL where the code used to generate the dataset may be found.                                                                                                                                          |
| Container    | OPTIONAL              | [object][]    | Used to specify the location and relevant attributes of software container image used to produce the dataset. Valid keys in this object include `Type`, `Tag` and [`URI`][uri] with [string][] values. |

Example:

```JSON
{
  "Name": "The mother of all experiments",
  "BIDSVersion": "1.6.0",
  "DatasetType": "raw",
  "License": "CC0",
  "Authors": [
    "Paul Broca",
    "Carl Wernicke"
  ],
  "Acknowledgements": "Special thanks to Korbinian Brodmann for help in formatting this dataset in BIDS. We thank Alan Lloyd Hodgkin and Andrew Huxley for helpful comments and discussions about the experiment and manuscript; Hermann Ludwig Helmholtz for administrative support; and Claudius Galenus for providing data for the medial-to-lateral index analysis.",
  "HowToAcknowledge": "Please cite this paper: https://www.ncbi.nlm.nih.gov/pubmed/001012092119281",
  "Funding": [
    "National Institute of Neuroscience Grant F378236MFH1",
    "National Institute of Neuroscience Grant 5RMZ0023106"
  ],
  "EthicsApprovals": [
    "Army Human Research Protections Office (Protocol ARL-20098-10051, ARL 12-040, and ARL 12-041)"
  ],
  "ReferencesAndLinks": [
    "https://www.ncbi.nlm.nih.gov/pubmed/001012092119281",
    "Alzheimer A., & Kraepelin, E. (2015). Neural correlates of presenile dementia in humans. Journal of Neuroscientific Data, 2, 234001. doi:1920.8/jndata.2015.7"
  ],
  "DatasetDOI": "doi:10.0.2.3/dfjj.10",
  "HEDVersion": "8.0.0",
  "GeneratedBy": [
    {
      "Name": "reproin",
      "Version": "0.6.0",
      "Container": {
        "Type": "docker",
        "Tag": "repronim/reproin:0.6.0"
      }
    }
  ],
  "SourceDatasets": [
    {
      "URL": "s3://dicoms/studies/correlates",
      "Version": "April 11 2011"
    }
  ]
}
```

#### Derived dataset and pipeline description

As for any BIDS dataset, a `dataset_description.json` file MUST be found at the
top level of every derived dataset:
`<dataset>/derivatives/<pipeline_name>/dataset_description.json`.

In contrast to raw BIDS datasets, derived BIDS datasets MUST include a
`GeneratedBy` key:

{{ MACROS___make_metadata_table(
   {
      "GeneratedBy": "REQUIRED"
   }
) }}

If a derived dataset is stored as a subfolder of the raw dataset, then the `Name` field
of the first `GeneratedBy` object MUST be a substring of the derived dataset folder name.
That is, in a directory `<dataset>/derivatives/<pipeline>[-<variant>]/`, the first
`GeneratedBy` object should have a `Name` of `<pipeline>`.

Example:

```JSON
{
  "Name": "FMRIPREP Outputs",
  "BIDSVersion": "1.6.0",
  "DatasetType": "derivative",
  "GeneratedBy": [
    {
      "Name": "fmriprep",
      "Version": "1.4.1",
      "Container": {
        "Type": "docker",
        "Tag": "poldracklab/fmriprep:1.4.1"
        }
    },
    {
      "Name": "Manual",
      "Description": "Re-added RepetitionTime metadata to bold.json files"
    }
  ],
  "SourceDatasets": [
    {
      "DOI": "doi:10.18112/openneuro.ds000114.v1.0.1",
      "URL": "https://openneuro.org/datasets/ds000114/versions/1.0.1",
      "Version": "1.0.1"
    }
  ]
}
```

### `README`

Every BIDS dataset SHOULD come with a free form text file (`README`) describing the dataset in more detail.
The `README` file MUST be either in ASCII or UTF-8 encoding.
A guideline for creating a good `README` file can be found in the
[bids-starter-kit](https://github.com/bids-standard/bids-starter-kit/blob/master/templates/README).

### `CHANGES`

Version history of the dataset (describing changes, updates and corrections) MAY
be provided in the form of a `CHANGES` text file. This file MUST follow the
[CPAN Changelog convention](https://metacpan.org/pod/release/HAARG/CPAN-Changes-0.400002/lib/CPAN/Changes/Spec.pod).
The `CHANGES` file MUST be either in ASCII or UTF-8 encoding.

Example:

```Text
1.0.1 2015-08-27
  - Fixed slice timing information.

1.0.0 2015-08-17
  - Initial release.
```

### `LICENSE`

A `LICENSE` file MAY be provided in addition to the short specification of the
used license in the `dataset_description.json` `"License"` field.
The `"License"` field and `LICENSE` file MUST correspond.
The `LICENSE` file MUST be either in ASCII or UTF-8 encoding.

## Participants file

Template:

```Text
participants.tsv
participants.json
```

The purpose of this RECOMMENDED file is to describe properties of participants
such as age, sex, handedness, species and strain.
If this file exists, it MUST contain the column `participant_id`,
which MUST consist of `sub-<label>` values identifying one row for each participant,
followed by a list of optional columns describing participants.
Each participant MUST be described by one and only one row.

The RECOMMENDED `species` column SHOULD be a binomial species name from the
[NCBI Taxonomy](https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi)
(for examples `homo sapiens`, `mus musculus`, `rattus norvegicus`).
For backwards compatibility, if `species` is absent, the participant is assumed to be
`homo sapiens`.

Commonly used *optional* columns in `participants.tsv` files are `age`, `sex`,
`handedness`, `strain`, and `strain_rrid`. We RECOMMEND to make use
of these columns, and in case that you do use them, we RECOMMEND to use the
following values for them:

{{ MACROS___make_columns_table(
   {
      "participant_id": ("REQUIRED", "There MUST be exactly one row for each participant."),
      "species": "RECOMMENDED",
      "age": "RECOMMENDED",
      "sex": "RECOMMENDED",
      "handedness": "RECOMMENDED",
      "strain": "RECOMMENDED",
      "strain_rrid": "RECOMMENDED",
   }
) }}

Throughout BIDS you can indicate missing values with `n/a` (for "not
available").

`participants.tsv` example:

```Text
participant_id age sex handedness group
sub-01 34 M right read
sub-02 12 F right write
sub-03 33 F n/a read
```

It is RECOMMENDED to accompany each `participants.tsv` file with a sidecar
`participants.json` file to describe the TSV column names and properties of their values (see also
the [section on tabular files](02-common-principles.md#tabular-files)).
Such sidecar files are needed to interpret the data, especially so when
optional columns are defined beyond `age`, `sex`, `handedness`, `species`, `strain`,
and `strain_rrid`, such as `group` in this example, or when a different
age unit is needed (for example, gestational weeks).
If no `units` is provided for age, it will be assumed to be in years relative
to date of birth.

`participants.json` example:

```JSON
{
    "age": {
        "Description": "age of the participant",
        "Units": "years"
    },
    "sex": {
        "Description": "sex of the participant as reported by the participant",
        "Levels": {
            "M": "male",
            "F": "female"
        }
    },
    "handedness": {
        "Description": "handedness of the participant as reported by the participant",
        "Levels": {
            "left": "left",
            "right": "right"
        }
    },
    "group": {
        "Description": "experimental group the participant belonged to",
        "Levels": {
            "read": "participants who read an inspirational text before the experiment",
            "write": "participants who wrote an inspirational text before the experiment"
        }
    }
}
```

## Samples file

Template:

```Text
samples.tsv
samples.json
```

The purpose of this file is to describe properties of samples, indicated by the `sample` entity.
This file is REQUIRED if `sample-<label>` is present in any filename within the dataset.
Each sample MUST be described by one and only one row.

{{ MACROS___make_columns_table(
   {
      "sample_id": ("REQUIRED", "The combination of `sample_id` and `participant_id` MUST be unique."),
      "participant_id": ("REQUIRED", "The combination of `sample_id` and `participant_id` MUST be unique."),
      "sample_type": "REQUIRED",
      "pathology": "RECOMMENDED",
      "derived_from": "RECOMMENDED",
   }
) }}

`samples.tsv` example:

```Text
sample_id participant_id sample_type derived_from
sample-01 sub-01 tissue n/a
sample-02 sub-01 tissue sample-01
sample-03 sub-01 tissue sample-01
sample-04 sub-02 tissue n/a
sample-05 sub-02 tissue n/a
```

It is RECOMMENDED to accompany each `samples.tsv` file with a sidecar
`samples.json` file to describe the TSV column names and properties of their values
(see also the [section on tabular files](02-common-principles.md#tabular-files)).

`samples.json` example:

```JSON
{
    "sample_type": {
        "Description": "type of sample from ENCODE Biosample Type (https://www.encodeproject.org/profiles/biosample_type)",
    },
    "derived_from": {
        "Description": "sample_id from which the sample is derived"
    }
}
```

## Phenotypic and assessment data

Template:

```Text
phenotype/
    <measurement_tool_name>.tsv
    <measurement_tool_name>.json
```

Optional: Yes

If the dataset includes multiple sets of participant level measurements (for
example responses from multiple questionnaires) they can be split into
individual files separate from `participants.tsv`.

Each of the measurement files MUST be kept in a `/phenotype` directory placed
at the root of the BIDS dataset and MUST end with the `.tsv` extension.
File names SHOULD be chosen to reflect the contents of the file.
For example, the "Adult ADHD Clinical Diagnostic Scale" could be saved in a file
called `/phenotype/acds_adult.tsv`.

The files can include an arbitrary set of columns, but one of them MUST be
`participant_id` and the entries of that column MUST correspond to the subjects
in the BIDS dataset and `participants.tsv` file.

As with all other tabular data, the additional phenotypic information files
MAY be accompanied by a JSON file describing the columns in detail
(see [Tabular files](02-common-principles.md#tabular-files)).

In addition to the column descriptions, the JSON file MAY contain the following fields:

{{ MACROS___make_metadata_table(
   {
      "MeasurementToolMetadata": "OPTIONAL",
      "Derivative": "OPTIONAL",
   }
) }}

As an example, consider the contents of a file called
`phenotype/acds_adult.json`:

```JSON
{
  "MeasurementToolMetadata": {
    "Description": "Adult ADHD Clinical Diagnostic Scale V1.2",
    "TermURL": "https://www.cognitiveatlas.org/task/id/trm_5586ff878155d"
  },
  "adhd_b": {
    "Description": "B. CHILDHOOD ONSET OF ADHD (PRIOR TO AGE 7)",
    "Levels": {
      "1": "YES",
      "2": "NO"
    }
  },
  "adhd_c_dx": {
    "Description": "As child met A, B, C, D, E and F diagnostic criteria",
    "Levels": {
      "1": "YES",
      "2": "NO"
    }
  }
}
```

Please note that in this example `MeasurementToolMetadata` includes information
about the questionnaire and `adhd_b` and `adhd_c_dx` correspond to individual
columns.

In addition to the keys available to describe columns in all tabular files
(`LongName`, `Description`, `Levels`, `Units`, and `TermURL`) the
`participants.json` file as well as phenotypic files can also include column
descriptions with a `Derivative` field that, when set to true, indicates that
values in the corresponding column is a transformation of values from other
columns (for example a summary score based on a subset of items in a
questionnaire).

## Scans file

Template:

```Text
sub-<label>/
    [ses-<label>/]
        sub-<label>[_ses-<label>]_scans.tsv
        sub-<label>[_ses-<label>]_scans.json
```

Optional: Yes

The purpose of this file is to describe timing and other properties of each
imaging acquisition sequence (each *run* file) within one session.

Each neural recording *file* SHOULD be described by exactly one row.
Some recordings consist of multiple parts, that span several files,
for example through `echo-`, `part-`, or `split-` entities.
Such recordings MUST be documented with one row per file.

{{ MACROS___make_columns_table(
   {
      "filename": ("REQUIRED", "There MUST be exactly one row for each file."),
      "acq_time__scans": ("OPTIONAL"),
   }
) }}

Additional fields can include external behavioral measures relevant to the
scan.
For example vigilance questionnaire score administered after a resting
state scan.
All such included additional fields SHOULD be documented in an accompanying
`_scans.json` file that describes these fields in detail
(see [Tabular files](02-common-principles.md#tabular-files)).

Example `_scans.tsv`:

```Text
filename	acq_time
func/sub-control01_task-nback_bold.nii.gz	1877-06-15T13:45:30
func/sub-control01_task-motor_bold.nii.gz	1877-06-15T13:55:33
meg/sub-control01_task-rest_split-01_meg.nii.gz	1877-06-15T12:15:27
meg/sub-control01_task-rest_split-02_meg.nii.gz	1877-06-15T12:15:27
```

## Sessions file

Template:

```Text
sub-<label>/
    sub-<label>_sessions.tsv
```

Optional: Yes

In case of multiple sessions there is an option of adding additional
`sessions.tsv` files describing variables changing between sessions.
In such case one file per participant SHOULD be added.
These files MUST include a `session_id` column and describe each session by one and only one row.
Column names in `sessions.tsv` files MUST be different from group level participant key column names in the
[`participants.tsv` file](./03-modality-agnostic-files.md#participants-file).

{{ MACROS___make_columns_table(
   {
      "session_id": ("REQUIRED", "There MUST be exactly one row for each session."),
      "acq_time__sessions": ("OPTIONAL"),
      "pathology": "RECOMMENDED",
   }
) }}

`_sessions.tsv` example:

```Text
session_id	acq_time	systolic_blood_pressure
ses-predrug	2009-06-15T13:45:30	120
ses-postdrug	2009-06-16T13:45:30	100
ses-followup	2009-06-17T13:45:30	110
```

## Code

Template: `code/*`

Source code of scripts that were used to prepare the dataset MAY be stored here.
Examples include anonymization or defacing of the data, or
the conversion from the format of the source data to the BIDS format
(see [source vs. raw vs. derived data](./02-common-principles.md#source-vs-raw-vs-derived-data)).
Extra care should be taken to avoid including original IDs or
any identifiable information with the source code.
There are no limitations or recommendations on the language and/or
code organization of these scripts at the moment.

<!-- Link Definitions -->

[object]: https://www.json.org/json-en.html

[string]: https://www.w3schools.com/js/js_json_datatypes.asp

[uri]: ./02-common-principles.md#uniform-resource-indicator