Chopping files semi-automatically

[RayStick]

A lot of researchers will have one physiological file for the whole scan session, which will need to be chopped into separate files corresponding to each scan to be analyzed. Offering a way of semi-automatically chopping these files, and allowing phys2bids to output multiple physiological files from the same input file, would be useful.

[smoia]

This is a topic we briefly discussed at DCCC 2019 with @yarikoptic .
It would be a great enhancement!
However, we need to think about this in a smart way.
The way @yarikoptic was thinking about approaching it is by using timestamps. However, we don't have them by default in our files - so if interested in this option, we should definitely remember this for "good practices".

[RayStick]

I have some MATLAB code that does this semi-automatically (it uses triggers for the time stamps alongside user input). At a later date, I could attempt to adapt this for phys2bids, if we wanted it.

[RayStick]

An important related point to "chopping" files. I think there are two options:

1. The input file the user brings to phys2bids is already the part of the data that corresponds to one MRI scan (i.e. we don't offer a way of chopping files)
2. The input file the user brings to phys2bids can be a file that contains data from multiple scans, and phys2bids can chop it. If it does chop it, we also have to consider the user may want the time stamp to start X seconds before the beginning of the scan and end X seconds after. This padding is important for later analysis.

[smoia]

@tsalo might have an idea on how to do this!

[tsalo]

I haven't had a chance to finalize it, but my approach with BioPac involves the following:

0. Set up your BioPac profile so that segments are labeled with a timestamp and a high maximum segment duration (3 hours)
1. When recording, have one long segment instead of stopping between scans.
2. Load data and identify trigger periods (onset and duration).
3. Infer absolute trigger period onsets from relative onsets (in samples), sample rate, and event marker label (the timestamp).
4. Load non-anonymized acquisition datetimes from the BIDS dataset.
5. Compare physio onsets with scan onsets to assign physio recordings to scans.
   • My guess is that this will require a little room for mismatch between the clock on the stimulus computer and the one on the physio computer.
6. Infer scan offsets based on scan onset, number of volumes, and repetition time.
   • Not sure how well this generalizes across scans folks might record physio for.
   • The reason I'm leaning toward using the scans to determine offsets instead of the physio trigger periods is because the physio trigger may not be disabled if the task is quit early. An alternative could be to use the heuristic, of course.
7. Split and output physio data.

I don't think this would require user input beyond the heuristic file (for optional durations and settings regarding physio data before and after the scan to retain).

[RayStick]

I have some working MATLAB code which I use to chop up a file which contains multiple scans. I will explain what it does here, in case any part of it is appealing to adapt for phys2bids. I haven't thought about how this would interact with BIDS as @tsalo has done, though! So incorporating in some of his good ideas would make sense too.

I have a function that reads in the input text file, alongside other arguments relevant to that scan - trigger column, sampling frequency, number of time points, amount of extra time before and after the scan to export the physiological data, and a json file which reads in slice timing information(from a DICOM to NIFTI conversion). At the moment, I have the user clicking before and after the scan (Fig1) and then I find the closest trigger to their click to get the index for the start and end. The output summary (Fig2, and Fig3 - zoomed in) shows the user the time window that will be exported into a single file (with triggers binarized). The other plots in Fig2/3 relate to the fact that the user can ask to “make up” volume triggers and excite triggers.

Is this too much user interaction for chopping up files? I.e. Would we want some way of doing it completely automatically?

[smoia]

@RayStick Sorry i didn't see this earlier.

It's a bit of user interaction more than the optimal, and we have to be sure it's not LabChart specific.
We could still implement this as a side function, something that specific users can adopt to chop their files (provided that x, y, and z).

I was thinking about this issue as well, and I would like to see where #173 goes first, but an easy option would be to add a function that uses what is already in the code in an iterative way.
For instance, let's call the total amount of timepoints in the file tntp.
The user can give us the amount of timepoints in each acquisition ntp, contained in a list list_ntp. For each acquisition we also have tr, the TR of the acquisition, contained in a list list_tr.
We could:

1. count tntp as we're doing now
2. check that it matches with sum(list_ntp)
3. for each ntp, tr pair multiply them together, add a pad at the beginning and at the end, and chop the file into the relative chunk that should be equivalent to the inputs we have now
4. pass this chunk to phy2bids as it is now
5. repeat 3-4 deleting the beginning of the file until ntp*tr.

What do you think?

[sangfrois]

hey @smoia, glad to see there could be an elegant way of cutting physio files into chunks corresponding to fMRI acquisition.

Two things, I'm having difficulty figuring out how you'd want the users to give the amount of time points per acquisition (list_ntp) and at what level such a function should be implemented.

• Would you derive ntp from an argument ?
  • if so, would that argument be in seconds, then converted into time points ?
• Could you be a bit more specific as to where it should be implemented in the code, or where you would add this implementation.
  • Could you point to specific parts of the code that are already written and that could be used for this task ?
  • in physio_obj.py ? where's tntp ?

I'm missing some pieces of information here, and I thought it might be helpful for others to know as well.

[smoia]

Yes, sorry, I didn't mean we have tntp now, I meant that now we are counting the total amount of trigger points in the file (it's the num_timepoints_found attribute of the BlueprintInput object - but it's meant for single run files.
In the solution I'm proposing above, I was thinking that:

• We could change the parser so that it accepts lists for -ntp and -TR. In that case, maintaining the current input type, ntp would become a list of integers, while tr would become a list of floats. For instance, say your file has 3 runs, the first 100 timepoints at TR=1.2 s, the second 90 timepoints at TR=1.5 s, the third 50 timepoints at TR=3 s. The user could call phys2bids -ntp 100 90 50 -tr 1.2 1.5 3 [...].
• Within the code that @vinferrer merged today in Add automatic method to count the number of MRI triggers #183 we could check that we find sum(ntp) timepoints, and then reiterate it as described above to check where the chunks start and end.
• Toward the end of phys2bids.py, where we loop to create BlueprintOutput objects, we could add a loop that splits the file into chuncks.

I'm happy if you jump into this @sangfrois, but please note that this issue is already assigned to @RayStick, so you should at least ask her what is the status of this development and whether she wants to work on it together, transfer assignment, or keep developing it on her own.

[sangfrois]

Yeah, for sure. I mean : I was peeking at this issue since it is relevant to my project. I wasn't that invested into the idea of implementing solutions, but more into looking at how things were evolving here.

Sorry, @RayStick, my intention was not to intrude in your development or pressure in any way. But, please, feel free to pin me if you need anything. I'd be delighted to give my input if you think I can help.

Also, Thanks @smoia for digging deeper in your idea. I'm still at a level of studying how functions are built and how they work together. If I can manage to do that and Rachael is opened to my suggestions , I'll send updates here.

[RayStick]

@sangfrois thanks for your questions, as I was confused about a few of the same things.

@smoia - I like the idea you put forward on March 23rd. I think it could definitely work, and it doesn't requires a clunky user interaction, which is good. After steps 1-5 we could plot the data to show the user what scans have been found and how they have been 'chopped' in the file. As we have for the trigger threshold now, if this automatic chopping has not worked correctly, a manual chopping can be implemented. Maybe this manually chopping could be similar to the MATLAB code I explained.

@sangfrois - You are not intruding and we're definitely open to suggestions! :)

I think this is a really important feature however I don't think I will be able to work on it anytime soon. I am currently doing python tutorials - let's see how fast I can learn!

[smoia]

@RayStick would you like to work on this in combo, maybe with @sangfrois ?
I can help too - let's just assign it so that we know who's going to take care of it.

I'm adding it to the first non-beta milestone.

[sangfrois]

@RayStick happy to hear that my questions have been helpful! Hope you feel you're gaining terrain on python with these tutorials!!

@smoia sounds good to me!

[RayStick]

@smoia - I am happy to be assigned to it, and will do my best to complete this by the first non-beta milestone. By June seems feasible! I am just not certain when I'll start working on it yet, as I want to complete more python tutorials first. So I'm happy for any assistance/ideas, or if people want to get started on it sooner I'm okay with that too (Don't want to hold things up!).

[smoia]

The questions are two:

1. do we need it earlier? @sangfrois , do you think that you're gong to use this and do you want it done?
2. @RayStick, do you take the responsibility of doing it in time for OHBM?
   The reason I'm asking is that it's an important feature to feature (pun intended), and while I think people could work on it in this moment, I'm not sure we'll be able to do it later in time (especially with OHBM approaching and if/when we go back to normal routines - I'm sure we'll be concentrating on other things then)

[RayStick]

@smoia On reflection, I have actually decided to assign myself to some other easier issues before tackling something like this, as I think I just need to get used to interacting with phys2bids code more first (and python in general). Therefore, I have taken myself off this issue for now. I will still keep track of the progress and happy to be a PR reviewer for it.

[smoia]

Ok, thank you @RayStick !
@sangfrois , wanna jump in? I can help you with its development if you want!

[RayStick]

And depending on how quickly I make progress with other issues, I may be able to help more with this. But no promises for now. :)

[sangfrois]

Alright with if Rachael focuses on other things, and yes, I am going to use this for sure.

Though I am still trying to get around the plan you've put together in the past few comments, I am willing to put some effort into that and make sure we can deliver it in time.

Of course, I'll need your help and patience as well ( my git workflow is not totally on point yet 😅). I think we'll manage!!