Simple GPU tracking #576

CHrlS98 · 2022-04-13T19:29:45Z

Script to compute a probabilistic tractogram on the GPU. The ODF image and mask are interpolated using nearest-neighbor interpolation. The script also incorporates ideas from Ensemble Tractography. Given a list of maximum angles, a different angle drawn at random from the set will be used for each streamline.

Here's a log showing the execution time on a Tractoflow-processed HCP subject.

>>> scil_compute_short_tracks.py FODF_Metrics/100206__fodf.nii.gz \
... Local_Tracking_Mask/100206__local_tracking_mask.nii.gz \
... a_big_short_tracks_tractogram.trk -v -f --nt 2000000

INFO:root:Generated 2000000 seed positions in 9.58s.
INFO:root:  100000/2000000 streamlines generated
INFO:root:  200000/2000000 streamlines generated
INFO:root:  300000/2000000 streamlines generated
INFO:root:  400000/2000000 streamlines generated
INFO:root:  500000/2000000 streamlines generated
INFO:root:  600000/2000000 streamlines generated
INFO:root:  700000/2000000 streamlines generated
INFO:root:  800000/2000000 streamlines generated
INFO:root:  900000/2000000 streamlines generated
INFO:root: 1000000/2000000 streamlines generated
INFO:root: 1100000/2000000 streamlines generated
INFO:root: 1200000/2000000 streamlines generated
INFO:root: 1300000/2000000 streamlines generated
INFO:root: 1400000/2000000 streamlines generated
INFO:root: 1500000/2000000 streamlines generated
INFO:root: 1600000/2000000 streamlines generated
INFO:root: 1700000/2000000 streamlines generated
INFO:root: 1800000/2000000 streamlines generated
INFO:root: 1900000/2000000 streamlines generated
INFO:root: 2000000/2000000 streamlines generated
INFO:root:Tracked 1081556 streamlines in 16.56s.
INFO:root:Saved tractogram to a_big_short_tracks_tractogram.trk in 25.90s.
INFO:root:Total runtime of 53.86s

And a visualization of the output tractogram.

Because it runs on the GPU and uses pyopencl which is an optional dependency of scilpy, there is not much I can test with pytest apart from the --help command.

scil-admin · 2022-04-13T20:09:42Z

Build passed ! Good Job 🍻 !

scil-admin · 2022-04-13T20:21:22Z

Build passed ! Good Job 🍻 !

scil-admin · 2022-04-13T20:47:33Z

Build passed ! Good Job 🍻 !

scil-admin · 2022-04-13T21:24:47Z

Build passed ! Good Job 🍻 !

scil-admin · 2022-04-21T15:02:58Z

Build passed ! Good Job 🍻 !

EmmaRenauld · 2022-05-16T14:36:54Z

It could possibly be useful to us eventually in dwiml!

What is the reason for creating short streamlines only? Memory restriction, or that was actually your initial goal?
@ppoulin91, I have not ported your GPU tracking to dwiml yet. Maybe at some point we could sit together with Charles to see if your two ideas can be compatible.

CHrlS98 · 2022-05-16T14:59:57Z

@EmmaRenauld

What is the reason for creating short streamlines only? Memory restriction, or that was actually your initial goal?

I just needed short streamlines and didn't need backward tracking so I didn't take the time to implement it. But it is not a memory limitation; you can increase the max_length to 300 and it will work (ish, because there is no backward tracking so WM-seeded tracks will end in the WM).

But, I think I will mark the PR as WIP and eventually rename to scil_compute_local_tracking_gpu.py (or something), implement the backward pass and change default values for min and max length.

Then we would have:

scil_compute_local_tracking.py (dipy)
scil_compute_local_tracking_dev.py (python)
scil_compute_local_tracking_gpu.py (gpu)

For clarity, maybe we should rename scil_compute_local_tracking.py -> scil_compute_local_tracking_dipy.py.

EmmaRenauld · 2022-05-16T15:06:03Z

Are the args so very different? Couldn't it be in the same script _dev and add option '--use_gpu'? Then the initialisation would call another Propagator class?

CHrlS98 · 2022-05-16T15:28:41Z

A lot of options would not be supported though:

Interpolation method for the sh field and mask is only NN,
Runge-Kutta interpolation is not implemented,
Deterministic tracking is not available,
There are no sf_threshold (although maybe there should be) and sf_init_threshold,
Etc.

I feel like it would just make the _dev script harder to read and understand. I'd prefer having a separate script with adequate description describing what GPU tracking does and doesn't do.

Or I could support all the options the _dev script supports, but that's a bigger job because everything has to be translated into OpenCL code.

I'll see how GPU tracking would fit in our tracking code structure. The propagator is 100% GPU code, as is the dataset, but I will at least write a proper GPUTracker class in the tracker.py file, with a track method for launching the GPU tracking.

EmmaRenauld · 2022-05-16T16:12:16Z

Nice!

scil-admin · 2022-05-16T16:24:32Z

Build passed ! Good Job 🍻 !

mdesco

Minor help edit to do. Otherwise, just to leave a trace, I'm having an error running it.

pyopencl._cl.RuntimeError: clBuildProgram failed: BUILD_PROGRAM_FAILURE - clBuildProgram failed: BUILD_PROGRAM_FAILURE - clBuildProgram failed: BUILD_PROGRAM_FAILURE

Build on <pyopencl.Device 'GeForce GTX 1080' on 'NVIDIA CUDA' at 0x2ef8f10>:

:174:55: error: implicit conversion from address space "global" to address space "private" is not supported when passing to parameter of destination type
sh_to_sf(odf_sh, sh_to_sf_mat, is_first_step, vertices,
^~~~~~~~
:30:54: note: passing argument to parameter 'vertices' here
const bool is_first_step, const float* vertices,
^

(options: -I /home/local/USHERBROOKE/desm2239/my_env/scilpy_master/lib/python3.7/site-packages/pyopencl/cl)

mdesco · 2022-05-16T17:15:31Z

scripts/scil_compute_short_tracks.py

@@ -0,0 +1,161 @@
+#!/usr/bin/env python3
+"""
+Perform probabilistic short-tracks tractography[1] on a ODF field inside a


Add space before [1] please

scil-admin · 2022-05-16T18:17:17Z

Build passed ! Good Job 🍻 !

scil-admin · 2022-05-16T18:22:53Z

Build passed ! Good Job 🍻 !

mdesco

Watatatow! Ça torpille cette affaire! npv10 en 155 seconds.
Bravo @CHrlS98 !!!

arnaudbore

Appart from the two comments I would say Great Job @CHrlS98
Maybe it need more testing but otherwise LGTM

scilpy/tracking/short_tracks.py

scripts/scil_compute_short_tracks.py

scil-admin · 2022-05-19T19:45:11Z

Build passed ! Good Job 🍻 !

scil-admin · 2022-05-19T20:21:34Z

Build passed ! Good Job 🍻 !

scil-admin · 2022-05-19T20:30:18Z

Build passed ! Good Job 🍻 !

CHrlS98 added 10 commits April 13, 2022 12:24

WIP GPU short tracks generator

701ce54

Add interpolation and is_valid_pos to CL kernel

54d00f9

Parallel GPU tracking

53d3a71

Better doc and verbose output

0dad5d8

Draw maximum angle from distribution

98c318f

Revert changes in io.image

0368c5c

Bugfix for single max angle

f2d5ef4

Add test for help arg

8731588

Update docstrings in opencl_utils.py

a95bc50

Removed step_size from input buffers

a9eccb3

scil-admin requested a review from arnaudbore April 13, 2022 20:09

Better random distribution for theta

b65ead6

Reformat help

122b944

CHrlS98 changed the title ~~Compute short-tracks tractogram (GPU tracking!)~~ [WIP] Compute short-tracks tractogram (GPU tracking!) May 16, 2022

mdesco reviewed May 16, 2022

View reviewed changes

CHrlS98 added 2 commits May 16, 2022 13:39

Add missing addess space qualifier

9017b45

Minor help edit

70743f4

mdesco approved these changes May 18, 2022

View reviewed changes

arnaudbore reviewed May 19, 2022

View reviewed changes

scilpy/tracking/short_tracks.py Outdated Show resolved Hide resolved

scripts/scil_compute_short_tracks.py Outdated Show resolved Hide resolved

CHrlS98 added 2 commits May 19, 2022 15:07

Add GPU tracker to tracking module and backward track.

3bc96fe

Update tests

6e02949

CHrlS98 changed the title ~~[WIP] Compute short-tracks tractogram (GPU tracking!)~~ Simple GPU tracking May 19, 2022

CHrlS98 added 2 commits May 19, 2022 15:20

Add todo to implement sf_threshold

7a6ce7a

Better doc for gpu tracking

75d92bc

scil-admin requested a review from arnaudbore May 19, 2022 19:45

arnaudbore approved these changes May 19, 2022

View reviewed changes

arnaudbore merged commit b162b8b into scilus:master May 19, 2022

CHrlS98 deleted the short-tracks-gpu branch May 19, 2022 20:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simple GPU tracking #576

Simple GPU tracking #576

CHrlS98 commented Apr 13, 2022 •

edited

Loading

scil-admin commented Apr 13, 2022

scil-admin commented Apr 13, 2022

scil-admin commented Apr 13, 2022

scil-admin commented Apr 13, 2022

scil-admin commented Apr 21, 2022

EmmaRenauld commented May 16, 2022

CHrlS98 commented May 16, 2022

EmmaRenauld commented May 16, 2022

CHrlS98 commented May 16, 2022

EmmaRenauld commented May 16, 2022

scil-admin commented May 16, 2022

mdesco left a comment

mdesco May 16, 2022

scil-admin commented May 16, 2022

scil-admin commented May 16, 2022

mdesco left a comment

arnaudbore left a comment

scil-admin commented May 19, 2022

scil-admin commented May 19, 2022

scil-admin commented May 19, 2022

Simple GPU tracking #576

Simple GPU tracking #576

Conversation

CHrlS98 commented Apr 13, 2022 • edited Loading

scil-admin commented Apr 13, 2022

scil-admin commented Apr 13, 2022

scil-admin commented Apr 13, 2022

scil-admin commented Apr 13, 2022

scil-admin commented Apr 21, 2022

EmmaRenauld commented May 16, 2022

CHrlS98 commented May 16, 2022

EmmaRenauld commented May 16, 2022

CHrlS98 commented May 16, 2022

EmmaRenauld commented May 16, 2022

scil-admin commented May 16, 2022

mdesco left a comment

Choose a reason for hiding this comment

mdesco May 16, 2022

Choose a reason for hiding this comment

scil-admin commented May 16, 2022

scil-admin commented May 16, 2022

mdesco left a comment

Choose a reason for hiding this comment

arnaudbore left a comment

Choose a reason for hiding this comment

scil-admin commented May 19, 2022

scil-admin commented May 19, 2022

scil-admin commented May 19, 2022

CHrlS98 commented Apr 13, 2022 •

edited

Loading