Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Function to copy derivatives #171

Merged
merged 36 commits into from
Apr 17, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
3c08d04
Added function to generate derivatives
Feb 19, 2021
a9ce261
refactor get_metadata
Remi-Gau Feb 27, 2021
f363672
reorganize dependencies
Remi-Gau Feb 27, 2021
3b6ff80
adapted query for metadata
Feb 24, 2021
0a541ca
fix test and lint
Remi-Gau Feb 27, 2021
59eb09e
remove dead code
Remi-Gau Feb 27, 2021
d502a8e
minor fixes on get_metadata
Remi-Gau Feb 28, 2021
2e00672
rename functions to use snake_case
Remi-Gau Feb 28, 2021
e20e967
refactor dependency indexing
Remi-Gau Feb 28, 2021
e0d12ba
get rid of parse_perf
Remi-Gau Feb 28, 2021
698251d
remove commented out dead code
Remi-Gau Feb 28, 2021
e51ac08
add comments
Remi-Gau Feb 28, 2021
28e7ee0
remove extra IF block
Remi-Gau Apr 16, 2021
e41e762
ASL jpg file handled like other dependency files
Remi-Gau Apr 16, 2021
d26c104
update ASL query tests to make sure context file dependency is indexed
Remi-Gau Apr 16, 2021
7352e6b
improving parsing of prefixes
Remi-Gau Apr 16, 2021
2ab00b6
silence get_metadata tests
Remi-Gau Apr 16, 2021
ed237ca
readd subfunction deleted during rebase
Remi-Gau Apr 16, 2021
1c6e897
reimplement #195
Remi-Gau Apr 16, 2021
2fb7b15
linting
Remi-Gau Apr 16, 2021
a63de7c
linting
Remi-Gau Apr 16, 2021
51d3023
rename derivate function
Remi-Gau Apr 16, 2021
618f2d4
move files
Remi-Gau Apr 17, 2021
e1d1c8a
reimplement getting metadata from files in root folder
Remi-Gau Apr 17, 2021
1baa5d9
add test to copy_to_derivative
Remi-Gau Apr 17, 2021
bcb7f1f
add instructions to get test data
Remi-Gau Apr 17, 2021
a01234b
filters for query can be passed as a n X 2 cell
Remi-Gau Apr 17, 2021
4c9d7b9
add test for copy_to_derivatives on real data
Remi-Gau Apr 17, 2021
7f0d9b1
add option to unzip and overwrite when copying to derivatives
Remi-Gau Apr 17, 2021
c7b3853
move test data for copy_to_derivatives
Remi-Gau Apr 17, 2021
997b07f
add test to check fmap dependencies are copied
Remi-Gau Apr 17, 2021
9581fa5
simplify tests copy derivatives
Remi-Gau Apr 17, 2021
0521460
move download moae function
Remi-Gau Apr 17, 2021
7a242ea
update github action
Remi-Gau Apr 17, 2021
b425fdb
update download moae
Remi-Gau Apr 17, 2021
63a9b1b
fix typo
Remi-Gau Apr 17, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions +bids/+internal/append_to_layout.m
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
function subject = append_to_layout(file, subject, modality, schema)
function [subject, p] = append_to_layout(file, subject, modality, schema)
%
% appends a file to the BIDS layout by parsing it according to the provided schema
%
Expand Down Expand Up @@ -40,7 +40,7 @@
p = bids.internal.parse_filename(file, entities);

% do not index json files when using the schema
if ~isempty(p) && strcmp(p.ext, '.json')
if isempty(p) || (~isempty(p) && strcmp(p.ext, '.json'))
return
end

Expand Down
23 changes: 23 additions & 0 deletions +bids/+internal/ends_with.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
function res = ends_with(str, pattern)
%
% Checks id character array 'str' ends with 'pat'
%
% USAGE res = bids.internal.endsWith(str, pat)
%
% str - character array
% pat - character array
%
% __________________________________________________________________________
%
% Based on spm_file.m and spm_select.m from SPM12.
% __________________________________________________________________________

% Copyright (C) 2011-2018 Guillaume Flandin, Wellcome Centre for Human Neuroimaging
res = false;
l_pat = length(pattern);
if l_pat > length(str)
return
end
res = strcmp(str(end - l_pat + 1:end), pattern);

end
92 changes: 92 additions & 0 deletions +bids/+internal/get_meta_list.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
function metalist = get_meta_list(filename, pattern)
%
% Read a BIDS's file metadata according to the inheritance principle
%
% USAGE::
%
% meta = bids.internal.get_metadata(filename, pattern = '^.*%s\\.json$')
%
% :param filename: fullpath name of file following BIDS standard
% :type filename: string
% :param pattern: Regular expression matching the metadata file (default is ``'^.*%s\\.json$'``)
% If provided, it must at least be ``'%s'``.
% :type pattern: string
%
%
% metalist - list of paths to metafiles
% __________________________________________________________________________

% Copyright (C) 2016-2018, Guillaume Flandin, Wellcome Centre for Human Neuroimaging
% Copyright (C) 2018--, BIDS-MATLAB developers

if nargin == 1
pattern = '^.*%s\\.json$';
end

pth = fileparts(filename);
p = bids.internal.parse_filename(filename);
metalist = {};

% Default assumes we are dealing with a file in the root directory
% like "participants.tsv"
% If the file has underscore separated entities ("sub-01_T1w.nii")
% then we look through the hierarchy for potential metadata file associated
% with queried file.
N = 1;
if isfield(p, 'entities')
N = 3;
% -There is a session level in the hierarchy
if isfield(p.entities, 'ses') && ~isempty(p.entities.ses)
N = N + 1;
end
end

for n = 1:N

% List the potential metadata files associated with this file suffix type
% Default is to assume it is a JSON file
metafile = bids.internal.file_utils('FPList', pth, sprintf(pattern, p.suffix));

if isempty(metafile)
metafile = {};
else
metafile = cellstr(metafile);
end

% For all those files we find which one is potentially associated with
% the file of interest
for i = 1:numel(metafile)

p2 = bids.internal.parse_filename(metafile{i});
entities = {};
if isfield(p2, 'entities')
entities = fieldnames(p2.entities);
end

% Check if this metadata file contains
% - the same entity-label pairs
% - same suffix
% as its data file counterpart
ismeta = true;
if ~strcmp(p.suffix, p2.suffix)
ismeta = false;
end
for j = 1:numel(entities)
if ~isfield(p.entities, entities{j}) || ...
~strcmp(p.entities.(entities{j}), p2.entities.(entities{j}))
ismeta = false;
break
end
end

% append path to list
if ismeta
metalist{end + 1, 1} = metafile{i}; %#ok<AGROW>
end

end

% Go up to the parent folder
pth = fullfile(pth, '..');
end
end
91 changes: 14 additions & 77 deletions +bids/+internal/get_metadata.m
Original file line number Diff line number Diff line change
@@ -1,101 +1,38 @@
function meta = get_metadata(filename, pattern)
function meta = get_metadata(metafile)
%
% Read a BIDS's file metadata according to the inheritance principle
%
% USAGE::
%
% meta = bids.internal.get_metadata(filename, pattern = '^.*%s\\.json$')
% meta = bids.internal.get_metadata(metafile)
%
% :param filename: fullpath name of file following BIDS standard
% :type filename: string
% :param pattern: Regular expression matching the metadata file (default is ``'^.*%s\\.json$'``)
% If provided, it must at least be ``'%s'``.
% :type pattern: string
% :param metafile: list of fullpath names of metadata files.
% :type metafile: string or array of strings
%
% :returns: - :meta: metadata structure
%
% meta - metadata structure
% .. todo
%
% add explanation on how the inheritance principle is implemented.
% __________________________________________________________________________

% Copyright (C) 2016-2018, Guillaume Flandin, Wellcome Centre for Human Neuroimaging
% Copyright (C) 2018--, BIDS-MATLAB developers

% assume most files are of the form *_suffix.json
% add an exception for files with no suffix, like participants.tsv
if nargin == 1
pattern = '^.*_?%s\\.json$';
end

pth = fileparts(filename);
p = bids.internal.parse_filename(filename);

meta = struct();
metafile = cellstr(metafile);

N = 3;

% -There is a session level in the hierarchy
if isfield(p, 'entities') && isfield(p.entities, 'ses') && ~isempty(p.entities.ses)
N = N + 1;
end

% -Loop from the directory where the file of interest is back to the
% top level of the BIDS hierarchy
for n = 1:N

% -List the potential metadata files associated with this file suffix type
% Default is to assume it is a JSON file
metafile = bids.internal.file_utils('FPList', pth, sprintf(pattern, p.suffix));

if isempty(metafile)
metafile = {};
for i = 1:numel(metafile)
if bids.internal.ends_with(metafile{i}, '.json')
meta = update_metadata(meta, bids.util.jsondecode(metafile{i}), metafile{i});
else
metafile = cellstr(metafile);
end

% -For all those files we find which one is potentially associated with
% the file of interest
for i = 1:numel(metafile)

p2 = bids.internal.parse_filename(metafile{i});
entities = {};
if isfield(p2, 'entities')
entities = fieldnames(p2.entities);
end

% Check if this metadata file contains
% - the same entity-label pairs
% - same prefix
% as its data file counterpart
ismeta = true;
if ~strcmp(p.suffix, p2.suffix)
ismeta = false;
end
for j = 1:numel(entities)
if ~isfield(p.entities, entities{j}) || ...
~strcmp(p.entities.(entities{j}), p2.entities.(entities{j}))
ismeta = false;
break
end
end

% -Read the content of the metadata file if it is a JSON file and update
% the metadata concerning the file of interest otherwise store the filename
if ismeta
if strcmp(p2.ext, '.json')
meta = update_metadata(meta, bids.util.jsondecode(metafile{i}), metafile{i});
else
meta.filename = metafile{i};
end
end

meta.filename = metafile{i};
end

% -Go up to the parent folder
pth = fullfile(pth, '..');

end

if isempty(meta)
warning('No metadata for %s', filename);
warning('No metadata for %s', metafile);
end

end
Expand Down
12 changes: 10 additions & 2 deletions +bids/+internal/parse_filename.m
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,16 @@
end

% identidy an eventual prefix to the file
tmp = regexp(parts{1}, '(sub)', 'split');
p.prefix = tmp{1};
% and amends the sub entity accordingly
p.prefix = '';
if strfind(parts{1}, 'sub-')
tmp = regexp(parts{1}, '(sub-)', 'split');
p.prefix = tmp{1};
if ~isempty(p.prefix)
p.entities.sub = p.entities.([p.prefix 'sub']);
p.entities = rmfield(p.entities, [p.prefix 'sub']);
end
end

% -Extra fields can be added to the structure and ordered specifically.
if nargin == 2
Expand Down
23 changes: 23 additions & 0 deletions +bids/+internal/starts_with.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
function res = starts_with(str, pattern)
%
% Checks id character array 'str' starts with 'pat'
%
% USAGE res = bids.internal.startsWith(str, pat)
%
% str - character array
% pat - character array
%
% __________________________________________________________________________
%
% Based on spm_file.m and spm_select.m from SPM12.
% __________________________________________________________________________

% Copyright (C) 2011-2018 Guillaume Flandin, Wellcome Centre for Human Neuroimaging
res = false;
l_pat = length(pattern);
if l_pat > length(str)
return
end
res = strcmp(str(1:l_pat), pattern);

end
Loading