Spacing transform not applied to dicom multi-frame image (transform for video frame/motion interpolation?)

[ChristianEschen]

It does not look like the spacing (resampling) transform is applied to dicom multi-frame images.
The image can be downloaded here: https://www.rubomedical.com/dicom_files/dicom_viewer_0002.zip

My environment is:
Ubuntu 18.04.5 LTS
Anaconda3: python 3.7.8
Monai: 0.5.3
itk: 5.2.0

Code to reproduce:

import monai
from monai.transforms import (
Compose,
LoadImaged,
Spacingd,
ToTensord)

dcm_path = '/home/gandalf/MIA/data/angio/2/0002.DCM'
label = 1
data_dict = [{'image': dcm_path, 'label': label}]

loader = LoadImaged(keys="image")
data_dict = loader(data_dict[0])
print(f"image shape: {data_dict['image'].shape}")
print(f"image spacing:\n{data_dict['image_meta_dict']['spacing']}")
print(f"image affine:\n{data_dict['image_meta_dict']['affine']}")

spacing = Spacingd(keys="image", pixdim=(
1.5, 1.5, 10.0), mode="bilinear")

data_dict = spacing(data_dict)
print(f"image shape: {data_dict['image'].shape}")
print(f"image spacing af Spacing:\n{data_dict['image_meta_dict']['spacing']}")
print(f"image affine after Spacing:\n{data_dict['image_meta_dict']['affine']}")

Output::

image spacing:
[ 1. 1. 33.]
image affine:
[[ 1. 0. 0. 0.]
[ 0. 1. 0. 0.]
[ 0. 0. 33. 0.]
[ 0. 0. 0. 1.]]
image shape: (96, 342, 342)
image spacing after Spacing:
[ 1. 1. 33.]
image affine after Spacing:
[[ 1.5 0. 0. 0. ]
[ 0. 1.5 0. 0. ]
[ 0. 0. 33. 0. ]
[ 0. 0. 0. 1. ]]

##############################
In the output above the first dimension is not resampled as expected.
I am not sure what the expected behavior should be for multi-frame images (vidoes), but I think this could be a potential bug or missing feature.

[wyli]

thanks for the question. the spacing transform assumes the input image has a "channel" dimension and the channel dimension should be the first (shape [channels, spatial_size_0, spaital_size_1,...]), so I think the current input (96, 342, 342) is interpreted as 96-channel 342x342 images. please try adding a EnsureChannelFirstD transform after LoadImageD or manually convert the image shape into (1, 96, 342, 342).

[ChristianEschen]

Thanks for the answer.
This does not help. It looks like there is an issue still:

import monai
from monai.transforms import (
Compose,
LoadImaged,
Spacingd,
ToTensord,
EnsureChannelFirstD)

dcm_path = '/home/gandalf/MIA/data/angio/2/0002.DCM'
label = 1
data_dict = [{'image': dcm_path, 'label': label}]

loader = LoadImaged(keys="image")

data_dict = loader(data_dict[0])
print(f"image shape: {data_dict['image'].shape}")
print(f"image spacing:\n{data_dict['image_meta_dict']['spacing']}")
print(f"image affine:\n{data_dict['image_meta_dict']['affine']}")

channel_first = EnsureChannelFirstD(keys="image")
data_dict = channel_first(data_dict)
print(f"image shape: {data_dict['image'].shape}")

spacing = Spacingd(keys="image", pixdim=(
1.5, 1.5, 10), mode="bilinear")

data_dict = spacing(data_dict)
print(f"image shape after Spacing: {data_dict['image'].shape}")
print(f"image spacing after Spacing:\n{data_dict['image_meta_dict']['spacing']}")
print(f"image affine after Spacing:\n{data_dict['image_meta_dict']['affine']}")

####output:
image shape: (96, 512, 512)
image spacing:
[ 1. 1. 33.]
image affine:
[[ 1. 0. 0. 0.]
[ 0. 1. 0. 0.]
[ 0. 0. 33. 0.]
[ 0. 0. 0. 1.]]
image shape: (1, 96, 512, 512)
image shape after Spacing: (1, 64, 342, 1687)
image spacing after Spacing:
[ 1. 1. 33.]
image affine after Spacing:
[[ 1.5 0. 0. 0. ]
[ 0. 1.5 0. 0. ]
[ 0. 0. 10. 0. ]
[ 0. 0. 0. 1. ]]

[wyli]

the coordinate system is only tracked using the affine, instead of the spacing field. I think the result is expected here:
input (1, 96, 512, 512), spacing (1, 1, 33) => output (1, 64, 342, 1687), spacing (1.5, 1.5, 10), please note the affine is updated.

[ChristianEschen]

thanks for the answer again.
Maybe the results are expected. Do you know how if it is possible toresample this volume (video) to pixel size 1.5, 1.5 and 10ms pr frame. The spacing in the third dimension here is referring to [00,18, 1063] as the ms/pr frame. (https://dicom.innolitics.com/ciods/us-multi-frame-image/cine/00181063)

[wyli]

ok, thanks, the transform is mainly for spatial interpolations, video frame/motion interpolation will not be supported here... I'm converting this to a discussion, perhaps @charliebudd @rijobro have som other comments or feature requests

[rijobro]

Since we don't currently support video transforms, perhaps it would be best to create your own transforms using something like OpenCV functionality?

(As an aside, is modifying the frame rate actually necessary for you? f you're treating your video on a frame-by-frame basis, then the frame rate isn't so important during the inference phase. Perhaps you have a justified need to modify the frame rate, I just wanted to check!)