From b298321c4492cc3d2211f75547fd85958bd2201f Mon Sep 17 00:00:00 2001 From: CPernet Date: Wed, 13 Sep 2023 16:06:19 +0200 Subject: [PATCH 01/14] add descriptions.tsv --- src/derivatives/common-data-types.md | 52 +++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index cf149c42b6..785cadc94f 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -252,7 +252,57 @@ A guide for using macros can be found at All REQUIRED metadata fields coming from a derivative file’s source file(s) MUST be propagated to the JSON description of the derivative unless the processing makes them invalid (for example, if a source 4D image is averaged to create a single -static volume, a `RepetitionTime` property would no longer be relevant). +static volume, a `RepetitionTime` property would no longer be relevant). This creates +a trail of the computational steps performed. Because many steps can occur until clean +data and because it is NOT mandatory to save every file or step, the simpler `desc-preproc` +and `desc-proc` are available. + +## descriptions.tsv + +To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, containing at least two columns: `desc_id` and `description`. This file can be located at the root derivatives level or at the subject level (inheritance principle). + +`desc_id` contains all labels used in the [`desc entity`](https://bids-specification.readthedocs.io/en/stable/appendices/entities.html#desc), while `description` is a human-readable description of what was computed. Note that while it is helpful to document how files are generated, we see this as 'light provenance' (see [provenance BEP28](https://github.com/bids-standard/BEP028_BIDSprov) for full computational reproducibility). + +{{ MACROS___make_filetree_example( + { + "raw/": { + CHANGES + README + channels.tsv + dataset_description.tsv + participants.tsv + "sub-001": { + "eeg": { + "sub-001_task-listening_events.tsv": "", + "sub-001_task-listening_events.json": "", + "sub-001_task-listening_eeg.edf": "", + "sub-001_task-listening_eeg.json": "", + }, + }, + }, + "derivatives/": { + descriptions.tsv + "sub-001": { + "eeg": { + "sub-001_task-listening_desc-filt_eeg.edf": "", + "sub-001_task-listening_desc-filt_eeg.json": "", + "sub-001_task-listening_desc-filt+ds_eeg.edf": "", + "sub-001_task-listening_desc-filt+ds_eeg.json": "", + "sub-001_task-listening_desc-preproc_eeg.edf": "", + "sub-001_task-listening_desc-preproc_eeg.json": "", }, + }, + } + } +) }} + +`descriptions.tsv` + +| desc_id | description | +|------------------|-----------------------------------------------------------------------------------------------------------| +| filt | low-pass filtered at 30Hz | +| filt+ds | low-pass filtered at 30Hz,downsampled to 250Hz | +| preproc | low-pass filtered at 30Hz, downsampled to 250Hz and rereferenced to a common average reference | + From 88650693b14513cc0f6bbf76a5ef15c452ce0049 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 13 Sep 2023 14:08:12 +0000 Subject: [PATCH 02/14] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- src/derivatives/common-data-types.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 785cadc94f..ee0cd643e3 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -252,16 +252,16 @@ A guide for using macros can be found at All REQUIRED metadata fields coming from a derivative file’s source file(s) MUST be propagated to the JSON description of the derivative unless the processing makes them invalid (for example, if a source 4D image is averaged to create a single -static volume, a `RepetitionTime` property would no longer be relevant). This creates -a trail of the computational steps performed. Because many steps can occur until clean -data and because it is NOT mandatory to save every file or step, the simpler `desc-preproc` +static volume, a `RepetitionTime` property would no longer be relevant). This creates +a trail of the computational steps performed. Because many steps can occur until clean +data and because it is NOT mandatory to save every file or step, the simpler `desc-preproc` and `desc-proc` are available. ## descriptions.tsv -To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, containing at least two columns: `desc_id` and `description`. This file can be located at the root derivatives level or at the subject level (inheritance principle). +To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, containing at least two columns: `desc_id` and `description`. This file can be located at the root derivatives level or at the subject level (inheritance principle). -`desc_id` contains all labels used in the [`desc entity`](https://bids-specification.readthedocs.io/en/stable/appendices/entities.html#desc), while `description` is a human-readable description of what was computed. Note that while it is helpful to document how files are generated, we see this as 'light provenance' (see [provenance BEP28](https://github.com/bids-standard/BEP028_BIDSprov) for full computational reproducibility). +`desc_id` contains all labels used in the [`desc entity`](https://bids-specification.readthedocs.io/en/stable/appendices/entities.html#desc), while `description` is a human-readable description of what was computed. Note that while it is helpful to document how files are generated, we see this as 'light provenance' (see [provenance BEP28](https://github.com/bids-standard/BEP028_BIDSprov) for full computational reproducibility). {{ MACROS___make_filetree_example( { @@ -280,7 +280,7 @@ To keep a record of what has been done to the data, a `descriptions.tsv` file ca }, }, }, - "derivatives/": { + "derivatives/": { descriptions.tsv "sub-001": { "eeg": { From 7def91ab8aa6ef1935ee2ea3d08f93a12e370555 Mon Sep 17 00:00:00 2001 From: Cyril Pernet Date: Fri, 15 Sep 2023 06:46:28 +0200 Subject: [PATCH 03/14] a little rewrite and removed +.md --- src/derivatives/common-data-types.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index ee0cd643e3..1e0603f7d3 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -252,10 +252,13 @@ A guide for using macros can be found at All REQUIRED metadata fields coming from a derivative file’s source file(s) MUST be propagated to the JSON description of the derivative unless the processing makes them invalid (for example, if a source 4D image is averaged to create a single -static volume, a `RepetitionTime` property would no longer be relevant). This creates -a trail of the computational steps performed. Because many steps can occur until clean -data and because it is NOT mandatory to save every file or step, the simpler `desc-preproc` -and `desc-proc` are available. +static volume, a `RepetitionTime` property would no longer be relevant). As each file +includes what was computed by increment, a trail of the computational steps performed +is created. Because many steps can occur until the fully clean data are obtained and +because it is NOT mandatory to save every file or step, the simpler `desc-preproc` +and `desc-proc` are available. It becomes however important to document what such +pre-processing or processing are, and record the order of computational steps. This +can be in the json file or alternatively described in a descriptions.tsv file. ## descriptions.tsv @@ -284,10 +287,10 @@ To keep a record of what has been done to the data, a `descriptions.tsv` file ca descriptions.tsv "sub-001": { "eeg": { - "sub-001_task-listening_desc-filt_eeg.edf": "", - "sub-001_task-listening_desc-filt_eeg.json": "", - "sub-001_task-listening_desc-filt+ds_eeg.edf": "", - "sub-001_task-listening_desc-filt+ds_eeg.json": "", + "sub-001_task-listening_desc-Filt_eeg.edf": "", + "sub-001_task-listening_desc-Filt_eeg.json": "", + "sub-001_task-listening_desc-FiltDs_eeg.edf": "", + "sub-001_task-listening_desc-FiltDs_eeg.json": "", "sub-001_task-listening_desc-preproc_eeg.edf": "", "sub-001_task-listening_desc-preproc_eeg.json": "", }, }, @@ -299,8 +302,8 @@ To keep a record of what has been done to the data, a `descriptions.tsv` file ca | desc_id | description | |------------------|-----------------------------------------------------------------------------------------------------------| -| filt | low-pass filtered at 30Hz | -| filt+ds | low-pass filtered at 30Hz,downsampled to 250Hz | +| Filt | low-pass filtered at 30Hz | +| FiltDs | low-pass filtered at 30Hz,downsampled to 250Hz | | preproc | low-pass filtered at 30Hz, downsampled to 250Hz and rereferenced to a common average reference | From 1ea7215916c648ab51172240d79fc0ddb7c89c1a Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 15 Sep 2023 04:46:48 +0000 Subject: [PATCH 04/14] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- src/derivatives/common-data-types.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 1e0603f7d3..4d6fa05b96 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -252,12 +252,12 @@ A guide for using macros can be found at All REQUIRED metadata fields coming from a derivative file’s source file(s) MUST be propagated to the JSON description of the derivative unless the processing makes them invalid (for example, if a source 4D image is averaged to create a single -static volume, a `RepetitionTime` property would no longer be relevant). As each file +static volume, a `RepetitionTime` property would no longer be relevant). As each file includes what was computed by increment, a trail of the computational steps performed -is created. Because many steps can occur until the fully clean data are obtained and +is created. Because many steps can occur until the fully clean data are obtained and because it is NOT mandatory to save every file or step, the simpler `desc-preproc` -and `desc-proc` are available. It becomes however important to document what such -pre-processing or processing are, and record the order of computational steps. This +and `desc-proc` are available. It becomes however important to document what such +pre-processing or processing are, and record the order of computational steps. This can be in the json file or alternatively described in a descriptions.tsv file. ## descriptions.tsv From a6a4986dc92da6a09d928bf4d54c40e57a4d977f Mon Sep 17 00:00:00 2001 From: Cyril Pernet Date: Fri, 15 Sep 2023 14:48:25 +0200 Subject: [PATCH 05/14] remove BEP 28 mention --- src/derivatives/common-data-types.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 4d6fa05b96..cc1d139922 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -264,7 +264,7 @@ can be in the json file or alternatively described in a descriptions.tsv file. To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, containing at least two columns: `desc_id` and `description`. This file can be located at the root derivatives level or at the subject level (inheritance principle). -`desc_id` contains all labels used in the [`desc entity`](https://bids-specification.readthedocs.io/en/stable/appendices/entities.html#desc), while `description` is a human-readable description of what was computed. Note that while it is helpful to document how files are generated, we see this as 'light provenance' (see [provenance BEP28](https://github.com/bids-standard/BEP028_BIDSprov) for full computational reproducibility). +`desc_id` contains all labels used in the [`desc entity`](https://bids-specification.readthedocs.io/en/stable/appendices/entities.html#desc), while `description` is a human-readable description of what was computed. Note that while it is helpful to document how files are generated, we see this as 'light provenance', i.e. it is not aimed at providing full computational reproducibility. {{ MACROS___make_filetree_example( { From 8b85864f75cf8f8694d5739c000f41130c28e441 Mon Sep 17 00:00:00 2001 From: Remi Gau Date: Fri, 15 Sep 2023 17:40:43 +0200 Subject: [PATCH 06/14] Update src/derivatives/common-data-types.md --- src/derivatives/common-data-types.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index cc1d139922..444428f934 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -301,10 +301,10 @@ To keep a record of what has been done to the data, a `descriptions.tsv` file ca `descriptions.tsv` | desc_id | description | -|------------------|-----------------------------------------------------------------------------------------------------------| -| Filt | low-pass filtered at 30Hz | +|------------------|-----------------------------------------------------------------------------------------------| +| Filt | low-pass filtered at 30Hz | | FiltDs | low-pass filtered at 30Hz,downsampled to 250Hz | -| preproc | low-pass filtered at 30Hz, downsampled to 250Hz and rereferenced to a common average reference | +| preproc | low-pass filtered at 30Hz, downsampled to 250Hz and rereferenced to a common average reference | From 0fed73b4b24d873d12d1ab752524dae0dbd533cb Mon Sep 17 00:00:00 2001 From: Remi Gau Date: Fri, 15 Sep 2023 17:42:37 +0200 Subject: [PATCH 07/14] Update src/derivatives/common-data-types.md --- src/derivatives/common-data-types.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 444428f934..6bf02eea55 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -300,13 +300,12 @@ To keep a record of what has been done to the data, a `descriptions.tsv` file ca `descriptions.tsv` -| desc_id | description | -|------------------|-----------------------------------------------------------------------------------------------| -| Filt | low-pass filtered at 30Hz | -| FiltDs | low-pass filtered at 30Hz,downsampled to 250Hz | +| desc_id | description | +|------------------|------------------------------------------------------------------------------------------------| +| Filt | low-pass filtered at 30Hz | +| FiltDs | low-pass filtered at 30Hz,downsampled to 250Hz | | preproc | low-pass filtered at 30Hz, downsampled to 250Hz and rereferenced to a common average reference | - [coordsys]: ../appendices/coordinate-systems.md#image-based-coordinate-systems From 03dce0a9bc60ec1b8b64b2f2beb3ec0e28a2e059 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Tue, 24 Oct 2023 18:51:43 +0200 Subject: [PATCH 08/14] Apply suggestions from code review Co-authored-by: Remi Gau --- src/derivatives/common-data-types.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 6bf02eea55..589e3c95a4 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -255,16 +255,16 @@ makes them invalid (for example, if a source 4D image is averaged to create a si static volume, a `RepetitionTime` property would no longer be relevant). As each file includes what was computed by increment, a trail of the computational steps performed is created. Because many steps can occur until the fully clean data are obtained and -because it is NOT mandatory to save every file or step, the simpler `desc-preproc` +because it is OPTIONAL to save every file or step, the simpler `desc-preproc` and `desc-proc` are available. It becomes however important to document what such pre-processing or processing are, and record the order of computational steps. This -can be in the json file or alternatively described in a descriptions.tsv file. +can be done in the JSON sidecar files or alternatively described in a `descriptions.tsv` file. ## descriptions.tsv -To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, containing at least two columns: `desc_id` and `description`. This file can be located at the root derivatives level or at the subject level (inheritance principle). +To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, containing at least two columns: `desc_id` and `description`. This file MAY be located at the root of the derivative dataset, or at the subject or session level ([Inheritance Principle](../common-principles.md#the-inheritance-principle))). -`desc_id` contains all labels used in the [`desc entity`](https://bids-specification.readthedocs.io/en/stable/appendices/entities.html#desc), while `description` is a human-readable description of what was computed. Note that while it is helpful to document how files are generated, we see this as 'light provenance', i.e. it is not aimed at providing full computational reproducibility. +`desc_id` contains all labels used in the [`desc entity`](../appendices/entities.md#desc), while `description` is a human-readable description of what was computed. Note that while it is helpful to document how files are generated, we see this as 'light provenance', i.e. it is not aimed at providing full computational reproducibility. {{ MACROS___make_filetree_example( { From 0b297ffe585c793a8cd625f926566a131eb7ec20 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Tue, 24 Oct 2023 18:58:17 +0200 Subject: [PATCH 09/14] rebase, fix remark and latin, nicer linebreaks --- src/derivatives/common-data-types.md | 37 +++++++++++++++++----------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 589e3c95a4..6a7ddd58d9 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -252,19 +252,28 @@ A guide for using macros can be found at All REQUIRED metadata fields coming from a derivative file’s source file(s) MUST be propagated to the JSON description of the derivative unless the processing makes them invalid (for example, if a source 4D image is averaged to create a single -static volume, a `RepetitionTime` property would no longer be relevant). As each file -includes what was computed by increment, a trail of the computational steps performed -is created. Because many steps can occur until the fully clean data are obtained and -because it is OPTIONAL to save every file or step, the simpler `desc-preproc` -and `desc-proc` are available. It becomes however important to document what such -pre-processing or processing are, and record the order of computational steps. This -can be done in the JSON sidecar files or alternatively described in a `descriptions.tsv` file. +static volume, a `RepetitionTime` property would no longer be relevant). +As each file includes what was computed by increment, +a trail of the computational steps performed is created. +Because many steps can occur until the fully clean data are obtained, +and because it is OPTIONAL to save every file or step, +the simpler `desc-preproc` and `desc-proc` are available. +It becomes however important to document what such pre-processing or processing are, +and record the order of computational steps. +This can be done in the JSON sidecar files or alternatively described in a `descriptions.tsv` file. ## descriptions.tsv -To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, containing at least two columns: `desc_id` and `description`. This file MAY be located at the root of the derivative dataset, or at the subject or session level ([Inheritance Principle](../common-principles.md#the-inheritance-principle))). +To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, +containing at least two columns: `desc_id` and `description`. +This file MAY be located at the root of the derivative dataset, +or at the subject or session level +([Inheritance Principle](../common-principles.md#the-inheritance-principle))). -`desc_id` contains all labels used in the [`desc entity`](../appendices/entities.md#desc), while `description` is a human-readable description of what was computed. Note that while it is helpful to document how files are generated, we see this as 'light provenance', i.e. it is not aimed at providing full computational reproducibility. +`desc_id` contains all labels used in the [`desc entity`](../appendices/entities.md#desc), +while `description` is a human-readable description of what was computed. +Note that while it is helpful to document how files are generated, we see this as *light provenance*, +that is, it is not aimed at providing full computational reproducibility. {{ MACROS___make_filetree_example( { @@ -300,11 +309,11 @@ To keep a record of what has been done to the data, a `descriptions.tsv` file ca `descriptions.tsv` -| desc_id | description | -|------------------|------------------------------------------------------------------------------------------------| -| Filt | low-pass filtered at 30Hz | -| FiltDs | low-pass filtered at 30Hz,downsampled to 250Hz | -| preproc | low-pass filtered at 30Hz, downsampled to 250Hz and rereferenced to a common average reference | +| desc_id | description | +|---------|------------------------------------------------------------------------------------------------| +| Filt | low-pass filtered at 30Hz | +| FiltDs | low-pass filtered at 30Hz,downsampled to 250Hz | +| preproc | low-pass filtered at 30Hz, downsampled to 250Hz and rereferenced to a common average reference | From 2a584e06ba88709e9f560b278d9828a81f64ca7f Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Tue, 7 Nov 2023 11:02:18 +0100 Subject: [PATCH 10/14] streamline language, fix errors --- src/derivatives/common-data-types.md | 78 +++++++++++++++++----------- 1 file changed, 48 insertions(+), 30 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 6a7ddd58d9..908d9d2138 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -264,36 +264,53 @@ This can be done in the JSON sidecar files or alternatively described in a `desc ## descriptions.tsv -To keep a record of what has been done to the data, a `descriptions.tsv` file can be used, -containing at least two columns: `desc_id` and `description`. +To keep a record of processing steps applied to the data, a `descriptions.tsv` file can be used. +The `descriptions.tsv` file MUST contain at least the following two columns: + +- `desc_id` +- `description` + This file MAY be located at the root of the derivative dataset, or at the subject or session level -([Inheritance Principle](../common-principles.md#the-inheritance-principle))). +([Inheritance Principle](../common-principles.md#the-inheritance-principle)). + +The `desc_id` column contains the labels used with the [`desc entity`](../appendices/entities.md#desc), +within the particular nesting that the `description.tsv` file is placed. +For example, if the `descriptions.tsv` file is placed at the root of the derivative dataset, +its `desc_id` column SHOULD contain all labels of the [`desc entity`](../appendices/entities.md#desc) +used across the entire derivative dataset. + +The `description` column contains human-readable descriptions of the processing steps. -`desc_id` contains all labels used in the [`desc entity`](../appendices/entities.md#desc), -while `description` is a human-readable description of what was computed. -Note that while it is helpful to document how files are generated, we see this as *light provenance*, -that is, it is not aimed at providing full computational reproducibility. +The use of `description.tsv` files together with the [`desc entity`](../appendices/entities.md#desc) +are helpful to document how files are generated, even if their use may not be sufficient +to provide full computational reproducibility. +### Example use of a `descriptions.tsv` file + + {{ MACROS___make_filetree_example( - { + { "raw/": { - CHANGES - README - channels.tsv - dataset_description.tsv - participants.tsv - "sub-001": { - "eeg": { - "sub-001_task-listening_events.tsv": "", - "sub-001_task-listening_events.json": "", - "sub-001_task-listening_eeg.edf": "", - "sub-001_task-listening_eeg.json": "", + "CHANGES": "", + "README": "", + "channels.tsv": "", + "dataset_description.tsv": "", + "participants.tsv": "", + "sub-001": { + "eeg": { + "sub-001_task-listening_events.tsv": "", + "sub-001_task-listening_events.json": "", + "sub-001_task-listening_eeg.edf": "", + "sub-001_task-listening_eeg.json": "", + }, }, }, - }, - "derivatives/": { - descriptions.tsv + "derivatives/": { + "descriptions.tsv": "", "sub-001": { "eeg": { "sub-001_task-listening_desc-Filt_eeg.edf": "", @@ -301,19 +318,20 @@ that is, it is not aimed at providing full computational reproducibility. "sub-001_task-listening_desc-FiltDs_eeg.edf": "", "sub-001_task-listening_desc-FiltDs_eeg.json": "", "sub-001_task-listening_desc-preproc_eeg.edf": "", - "sub-001_task-listening_desc-preproc_eeg.json": "", }, + "sub-001_task-listening_desc-preproc_eeg.json": "", + }, }, } - } + } ) }} -`descriptions.tsv` +Contents of the `descriptions.tsv` file: -| desc_id | description | -|---------|------------------------------------------------------------------------------------------------| -| Filt | low-pass filtered at 30Hz | -| FiltDs | low-pass filtered at 30Hz,downsampled to 250Hz | -| preproc | low-pass filtered at 30Hz, downsampled to 250Hz and rereferenced to a common average reference | +| desc_id | description | +|---------|-------------------------------------------------------------------------------------------------| +| Filt | low-pass filtered at 30Hz | +| FiltDs | low-pass filtered at 30Hz, downsampled to 250Hz | +| preproc | low-pass filtered at 30Hz, downsampled to 250Hz, and rereferenced to a common average reference | From 860d427030ec167b5d5b48b71c86cde8f4fc949c Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Tue, 7 Nov 2023 11:07:34 +0100 Subject: [PATCH 11/14] remove part of the language --- src/derivatives/common-data-types.md | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 908d9d2138..4d5f58389e 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -253,22 +253,14 @@ All REQUIRED metadata fields coming from a derivative file’s source file(s) MU be propagated to the JSON description of the derivative unless the processing makes them invalid (for example, if a source 4D image is averaged to create a single static volume, a `RepetitionTime` property would no longer be relevant). -As each file includes what was computed by increment, -a trail of the computational steps performed is created. -Because many steps can occur until the fully clean data are obtained, -and because it is OPTIONAL to save every file or step, -the simpler `desc-preproc` and `desc-proc` are available. -It becomes however important to document what such pre-processing or processing are, -and record the order of computational steps. -This can be done in the JSON sidecar files or alternatively described in a `descriptions.tsv` file. ## descriptions.tsv -To keep a record of processing steps applied to the data, a `descriptions.tsv` file can be used. +To keep a record of processing steps applied to the data, a `descriptions.tsv` file MAY be used. The `descriptions.tsv` file MUST contain at least the following two columns: -- `desc_id` -- `description` +- `desc_id` +- `description` This file MAY be located at the root of the derivative dataset, or at the subject or session level From caeeb0769de6230548039d0b540e3d261079ede4 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Tue, 7 Nov 2023 11:15:01 +0100 Subject: [PATCH 12/14] desc_id -> label --- src/derivatives/common-data-types.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 4d5f58389e..b8380169f9 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -259,17 +259,17 @@ static volume, a `RepetitionTime` property would no longer be relevant). To keep a record of processing steps applied to the data, a `descriptions.tsv` file MAY be used. The `descriptions.tsv` file MUST contain at least the following two columns: -- `desc_id` +- `label` - `description` This file MAY be located at the root of the derivative dataset, or at the subject or session level ([Inheritance Principle](../common-principles.md#the-inheritance-principle)). -The `desc_id` column contains the labels used with the [`desc entity`](../appendices/entities.md#desc), +The `label` column contains the labels used with the [`desc entity`](../appendices/entities.md#desc), within the particular nesting that the `description.tsv` file is placed. For example, if the `descriptions.tsv` file is placed at the root of the derivative dataset, -its `desc_id` column SHOULD contain all labels of the [`desc entity`](../appendices/entities.md#desc) +its `label` column SHOULD contain all labels of the [`desc entity`](../appendices/entities.md#desc) used across the entire derivative dataset. The `description` column contains human-readable descriptions of the processing steps. @@ -319,7 +319,7 @@ A guide for using macros can be found at Contents of the `descriptions.tsv` file: -| desc_id | description | +| label | description | |---------|-------------------------------------------------------------------------------------------------| | Filt | low-pass filtered at 30Hz | | FiltDs | low-pass filtered at 30Hz, downsampled to 250Hz | From 5ab8b1e34862f8a2fc9e8cc4ac5ae754c8b11e53 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Wed, 8 Nov 2023 11:08:34 +0100 Subject: [PATCH 13/14] fix: description -> descriptions (plural) --- src/derivatives/common-data-types.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index b8380169f9..2dd6cd6938 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -267,14 +267,14 @@ or at the subject or session level ([Inheritance Principle](../common-principles.md#the-inheritance-principle)). The `label` column contains the labels used with the [`desc entity`](../appendices/entities.md#desc), -within the particular nesting that the `description.tsv` file is placed. +within the particular nesting that the `descriptions.tsv` file is placed. For example, if the `descriptions.tsv` file is placed at the root of the derivative dataset, its `label` column SHOULD contain all labels of the [`desc entity`](../appendices/entities.md#desc) used across the entire derivative dataset. The `description` column contains human-readable descriptions of the processing steps. -The use of `description.tsv` files together with the [`desc entity`](../appendices/entities.md#desc) +The use of `descriptions.tsv` files together with the [`desc entity`](../appendices/entities.md#desc) are helpful to document how files are generated, even if their use may not be sufficient to provide full computational reproducibility. @@ -290,7 +290,7 @@ A guide for using macros can be found at "CHANGES": "", "README": "", "channels.tsv": "", - "dataset_description.tsv": "", + "dataset_description.json": "", "participants.tsv": "", "sub-001": { "eeg": { From fede5d3187f95c194cf3ff93bd780544a879b830 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Wed, 8 Nov 2023 11:11:32 +0100 Subject: [PATCH 14/14] label -> desc_id --- src/derivatives/common-data-types.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/derivatives/common-data-types.md b/src/derivatives/common-data-types.md index 2dd6cd6938..8ffc33c0cf 100644 --- a/src/derivatives/common-data-types.md +++ b/src/derivatives/common-data-types.md @@ -259,17 +259,17 @@ static volume, a `RepetitionTime` property would no longer be relevant). To keep a record of processing steps applied to the data, a `descriptions.tsv` file MAY be used. The `descriptions.tsv` file MUST contain at least the following two columns: -- `label` +- `desc_id` - `description` This file MAY be located at the root of the derivative dataset, or at the subject or session level ([Inheritance Principle](../common-principles.md#the-inheritance-principle)). -The `label` column contains the labels used with the [`desc entity`](../appendices/entities.md#desc), +The `desc_id` column contains the labels used with the [`desc entity`](../appendices/entities.md#desc), within the particular nesting that the `descriptions.tsv` file is placed. For example, if the `descriptions.tsv` file is placed at the root of the derivative dataset, -its `label` column SHOULD contain all labels of the [`desc entity`](../appendices/entities.md#desc) +its `desc_id` column SHOULD contain all labels of the [`desc entity`](../appendices/entities.md#desc) used across the entire derivative dataset. The `description` column contains human-readable descriptions of the processing steps. @@ -319,7 +319,7 @@ A guide for using macros can be found at Contents of the `descriptions.tsv` file: -| label | description | +| desc_id | description | |---------|-------------------------------------------------------------------------------------------------| | Filt | low-pass filtered at 30Hz | | FiltDs | low-pass filtered at 30Hz, downsampled to 250Hz |