From de4ab9f33c1c0b9f1b1cf6bee914e91a18ea6494 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 27 Jul 2020 10:31:37 +0200 Subject: [PATCH 01/10] add link rfc3339 --- src/02-common-principles.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 68d5da4cd0..1edf1255ca 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -459,7 +459,7 @@ pairs. JSON files MUST be in UTF-8 encoding. Extensive documentation of the format can be found here: [http://json.org/](http://json.org/). Several editors have built-in support for JSON syntax highlighting that aids manual creation of such files. An online editor for JSON with built-in validation is available at: -[http://jsoneditoronline.org](http://jsoneditoronline.org). +[http://jsoneditoronline.org](http://jsoneditoronline.org). It is RECOMMENDED that keys in a JSON file are written in [CamelCase](https://en.wikipedia.org/wiki/Camel_case) with the first letter in upper case (e.g., `SamplingFrequency`, not `samplingFrequency`). Note however, when a JSON file is used as an accompanying @@ -542,8 +542,8 @@ Describing dates and timestamps: - Date time information MUST be expressed in the following format `YYYY-MM-DDThh:mm:ss[.000000]` (year, month, day, hour (24h), minute, second, and optionally fractional second). - This is equivalent to the RFC3339 "date-time" format, time zone is always - assumed as local time). + This is equivalent to the [RFC3339](https://tools.ietf.org/html/rfc3339) + "date-time" format, time zone is always assumed as local time. No specific precision is required for fractional seconds, but the precision SHOULD be consistent across the dataset. For example `2009-06-15T13:45:30` From a9bf18ac87f13ba67a2d149ff103408a2d35fc1c Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 27 Jul 2020 13:02:33 +0200 Subject: [PATCH 02/10] refer to units section from modality specific --- .../02-magnetoencephalography.md | 8 ++------ .../03-electroencephalography.md | 8 ++------ .../04-intracranial-electroencephalography.md | 10 +++------- 3 files changed, 7 insertions(+), 19 deletions(-) diff --git a/src/04-modality-specific-files/02-magnetoencephalography.md b/src/04-modality-specific-files/02-magnetoencephalography.md index 269bae0c47..4d8e1e90b8 100644 --- a/src/04-modality-specific-files/02-magnetoencephalography.md +++ b/src/04-modality-specific-files/02-magnetoencephalography.md @@ -170,12 +170,8 @@ Example: ``` Note that the date and time information SHOULD be stored in the Study key file -(`scans.tsv`), see [Scans file](../03-modality-agnostic-files.md#scans-file). As -it is indicated there, date time information MUST be expressed in the following -format `YYYY-MM-DDThh:mm:ss` -([ISO8601](https://en.wikipedia.org/wiki/ISO_8601) date-time format). For -example: 2009-06-15T13:45:30. It does not need to be fully detailed, depending -on local REB/IRB ethics board policy. +(`scans.tsv`), see [Scans file](../03-modality-agnostic-files.md#scans-file). +Date time information MUST be expressed as indicated in [Units](../02-common-principles.md#units) ## Channels description (`*_channels.tsv`) diff --git a/src/04-modality-specific-files/03-electroencephalography.md b/src/04-modality-specific-files/03-electroencephalography.md index a6ee171195..64200f255b 100644 --- a/src/04-modality-specific-files/03-electroencephalography.md +++ b/src/04-modality-specific-files/03-electroencephalography.md @@ -183,12 +183,8 @@ Example: ``` Note that the date and time information SHOULD be stored in the Study key file -([`scans.tsv`](../03-modality-agnostic-files.md#scans-file)). As it is -indicated there, date time information MUST be expressed in the following -format `YYYY-MM-DDThh:mm:ss` -([ISO8601](https://en.wikipedia.org/wiki/ISO_8601) date-time format). For -example: 2009-06-15T13:45:30. It does not need to be fully detailed, depending -on local REB/IRB ethics board policy. +([`scans.tsv`](../03-modality-agnostic-files.md#scans-file)). +Date time information MUST be expressed as indicated in [Units](../02-common-principles.md#units) ## Channels description (`*_channels.tsv`) diff --git a/src/04-modality-specific-files/04-intracranial-electroencephalography.md b/src/04-modality-specific-files/04-intracranial-electroencephalography.md index 940825ed97..fb0c8fbfb6 100644 --- a/src/04-modality-specific-files/04-intracranial-electroencephalography.md +++ b/src/04-modality-specific-files/04-intracranial-electroencephalography.md @@ -200,12 +200,8 @@ Example: ``` Note that the date and time information SHOULD be stored in the Study key file -([`scans.tsv`](../03-modality-agnostic-files.md#scans-file)). As it is indicated -there, date time information MUST be expressed in the following format -`YYYY-MM-DDThh:mm:ss` -([ISO8601](https://en.wikipedia.org/wiki/ISO_8601) date-time format). For -example: 2009-06-15T13:45:30. It does not need to be fully detailed, depending -on local REB/IRB ethics board policy. +([`scans.tsv`](../03-modality-agnostic-files.md#scans-file)). +Date time information MUST be expressed as indicated in [Units](../02-common-principles.md#units) ## Channels description (`*_channels.tsv`) @@ -339,7 +335,7 @@ For example: The order of the required columns in the `*_electrodes.tsv` file MUST be as listed below. -MUST be present: +MUST be present: | Column name | Definition | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- | From ddf2d2ce87f8031280d613f4a3e771dc4e753dca Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 27 Jul 2020 13:03:06 +0200 Subject: [PATCH 03/10] add note on 'fully detailed' --- src/02-common-principles.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 1edf1255ca..8fdbeb9567 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -548,6 +548,12 @@ Describing dates and timestamps: SHOULD be consistent across the dataset. For example `2009-06-15T13:45:30` +- Note that depending on local REB/IRB ethics board policy, Date time information + does not need to be fully detailed. + For example, it is permissible to set the time to `00:00:00` if reporting the + exact recording time is undesirable. + However, see paragraph on anonymization below. + - Time stamp information MUST be expressed in the following format: `13:45:30[.000000]` From b06468c1cc2148767328f6e74ab33be1557b94d0 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Tue, 28 Jul 2020 11:35:38 +0200 Subject: [PATCH 04/10] add *almost* RFC3339 --- src/02-common-principles.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 8fdbeb9567..97e82195b6 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -542,8 +542,11 @@ Describing dates and timestamps: - Date time information MUST be expressed in the following format `YYYY-MM-DDThh:mm:ss[.000000]` (year, month, day, hour (24h), minute, second, and optionally fractional second). - This is equivalent to the [RFC3339](https://tools.ietf.org/html/rfc3339) - "date-time" format, time zone is always assumed as local time. + This is almost equivalent to the [RFC3339](https://tools.ietf.org/html/rfc3339) + "date-time" format, with the exception that in BIDS, time zone is always + assumed as local time. + The RFC3339 `Z` syntax to indicate UTC or the RFC3339 "offset" syntax are not + supported. No specific precision is required for fractional seconds, but the precision SHOULD be consistent across the dataset. For example `2009-06-15T13:45:30` From 843f276bdefad9fa08825ef92516736742402cef Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Sun, 2 Aug 2020 11:32:50 +0200 Subject: [PATCH 05/10] improve ordering, example, wording --- src/02-common-principles.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 97e82195b6..5a0acfa7a2 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -549,16 +549,18 @@ Describing dates and timestamps: supported. No specific precision is required for fractional seconds, but the precision SHOULD be consistent across the dataset. - For example `2009-06-15T13:45:30` + For example `2009-06-15T13:45:30`. + +- Time stamp information MUST be expressed in the following format: + `hh:mm:ss[.000000]` + For example `13:45:30`. - Note that depending on local REB/IRB ethics board policy, Date time information does not need to be fully detailed. For example, it is permissible to set the time to `00:00:00` if reporting the exact recording time is undesirable. - However, see paragraph on anonymization below. - -- Time stamp information MUST be expressed in the following format: - `13:45:30[.000000]` + However, for privacy protection reasons, it it generally RECOMMENDED to shift + dates, as described below. - Dates can be shifted by a random number of days for privacy protection reasons. From 4e59dd80dd956dc6165c9a26679f13a2c5efb000 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Sun, 2 Aug 2020 13:19:28 +0200 Subject: [PATCH 06/10] add warning about FIF recording dates --- src/02-common-principles.md | 5 +++++ src/99-appendices/06-meg-file-formats.md | 10 ++++++++++ 2 files changed, 15 insertions(+) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 5a0acfa7a2..19363aa1ac 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -570,6 +570,11 @@ Describing dates and timestamps: within each subject to maintain the interval information. For example: `1867-06-15T13:45:30` +- WARNING: The Neuromag/Elekta/MEGIN file format for MEG (`.fif`) does *not* + support recording dates earlier than `1902` roughly. + See [MEG-file-formats](./99-appendices/06-meg-file-formats.md#recording-dates-in-fif-files) + for more information. + - Age SHOULD be given as the number of years since birth at the time of scanning (or first scan in case of multi session datasets). Using higher accuracy (weeks) should in general be avoided due to privacy protection, diff --git a/src/99-appendices/06-meg-file-formats.md b/src/99-appendices/06-meg-file-formats.md index 733b4f4b29..54cc8fb4eb 100644 --- a/src/99-appendices/06-meg-file-formats.md +++ b/src/99-appendices/06-meg-file-formats.md @@ -118,6 +118,16 @@ More information can be found under the following links: - [Neuromag/Elekta/MEGIN data organization](http://www.fieldtriptoolbox.org/getting_started/neuromag) - [BabyMEG](http://www.fieldtriptoolbox.org/getting_started/babysquid) +### recording dates in `.fif` files + +It is important to note that recording dates in `.fif` files are represented +as `int32` format seconds since (or before) [*the Epoch*](https://en.wikipedia.org/wiki/Unix_time) +(`1970-01-01T00:00:00.000000` UTC). +Integers in `int32` format can encode values from -2,147,483,647 to +2,147,483,647. +Due to this representation, the Neuromag/Elekta/MEGIN file format for MEG (`.fif`) does *not* +support recording dates earlier than `1901-12-13T08:45:53.000000` UTC or later than +`2038-01-19T03:14:07.000000` UTC. + ## BTi/4D neuroimaging Each experimental run on a 4D neuroimaging/BTi system results in a folder From d11393f4b680b827ae4021b675c3defd086996fb Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Sun, 2 Aug 2020 17:43:26 +0200 Subject: [PATCH 07/10] Apply suggestions from code review Co-authored-by: Taylor Salo --- src/02-common-principles.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 19363aa1ac..d7f8a74241 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -555,12 +555,12 @@ Describing dates and timestamps: `hh:mm:ss[.000000]` For example `13:45:30`. -- Note that depending on local REB/IRB ethics board policy, Date time information +- Note that, depending on local REB/IRB ethics board policy, date time information does not need to be fully detailed. For example, it is permissible to set the time to `00:00:00` if reporting the exact recording time is undesirable. - However, for privacy protection reasons, it it generally RECOMMENDED to shift - dates, as described below. + However, for privacy protection reasons, it is generally RECOMMENDED to shift + dates, as described below, without completely removing time information, as time information can be useful for research purposes. - Dates can be shifted by a random number of days for privacy protection reasons. From 6ca1bcdb9a74d4bc6c2530e018623254ce49ea5a Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 3 Aug 2020 10:10:21 +0200 Subject: [PATCH 08/10] add optional UTC indicator --- src/02-common-principles.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index d7f8a74241..68ab46daba 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -540,13 +540,13 @@ For additional rules, see below: Describing dates and timestamps: - Date time information MUST be expressed in the following format - `YYYY-MM-DDThh:mm:ss[.000000]` (year, month, day, hour (24h), minute, - second, and optionally fractional second). + `YYYY-MM-DDThh:mm:ss[.000000][Z]` (year, month, day, hour (24h), minute, + second, optional fractional seconds, and optional UTC time indicator). This is almost equivalent to the [RFC3339](https://tools.ietf.org/html/rfc3339) - "date-time" format, with the exception that in BIDS, time zone is always - assumed as local time. - The RFC3339 `Z` syntax to indicate UTC or the RFC3339 "offset" syntax are not - supported. + "date-time" format, with the exception that UTC indicator `Z` is optional and + non-zero UTC offsets are not indicated. + If `Z` is not indicated, time zone is always assumed to be the local time of the + dataset viewer. No specific precision is required for fractional seconds, but the precision SHOULD be consistent across the dataset. For example `2009-06-15T13:45:30`. From 9f1e2e7de4facc2e2e89547995af0c1f6d3bcea7 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 3 Aug 2020 10:24:25 +0200 Subject: [PATCH 09/10] extend .fif warning --- src/02-common-principles.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 68ab46daba..2eec27c547 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -572,6 +572,9 @@ Describing dates and timestamps: - WARNING: The Neuromag/Elekta/MEGIN file format for MEG (`.fif`) does *not* support recording dates earlier than `1902` roughly. + Some analysis software packages (e.g., MNE-Python) handle their data as `.fif` + internally and will break if recording dates are specified prior to `1902`, + even if the original data format is not `.fif`. See [MEG-file-formats](./99-appendices/06-meg-file-formats.md#recording-dates-in-fif-files) for more information. From 362994827206b7256e91f9d3cd4f06b3479e1927 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Tue, 4 Aug 2020 19:51:18 +0200 Subject: [PATCH 10/10] Apply suggestions from code review Co-authored-by: Chris Markiewicz --- src/02-common-principles.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/src/02-common-principles.md b/src/02-common-principles.md index 2eec27c547..7443411b9f 100644 --- a/src/02-common-principles.md +++ b/src/02-common-principles.md @@ -555,12 +555,13 @@ Describing dates and timestamps: `hh:mm:ss[.000000]` For example `13:45:30`. -- Note that, depending on local REB/IRB ethics board policy, date time information - does not need to be fully detailed. +- Note that, depending on local ethics board policy, date time information may not + need to be fully detailed. For example, it is permissible to set the time to `00:00:00` if reporting the exact recording time is undesirable. - However, for privacy protection reasons, it is generally RECOMMENDED to shift - dates, as described below, without completely removing time information, as time information can be useful for research purposes. + However, for privacy protection reasons, it is RECOMMENDED to shift dates, as + described below, without completely removing time information, as time information + can be useful for research purposes. - Dates can be shifted by a random number of days for privacy protection reasons.