Does it support batch processing?

[ryandok]

A greast work!
I wonder if this work supports batch processing?
I used a 3D tensors "out" with the shape of [N, C, H, W, D], and class_idx=1,
and run map = cam_extractor(1, out),
bug got
File "/home/hra/anaconda3/envs/Pytorch1.7.1/lib/python3.7/site-packages/torchcam/methods/core.py", line 127, in _precheck raise ValueError(f"expected a 1-sized batch to be hooked. Received: {fmap.shape[0]}") ValueError: expected a 1-sized batch to be hooked. Received: 5
Would you please help me?
Thanks!

Best regards.
Ryan

[frgfm]

Hi @ryandok 👋

Of course, happy to help!
So I think I understand that your model is not a classification model? If that is the case, the problem is that under-the-hood, the out is expected to be of shape (N, K). So, the 3D is not a problem, but the task your model is designed for is. For instance, video or image classification models work, but because they will have the same number of output dimensions (2).

For better help, I would need the full snippet to reproduce this 🙏 And perhaps, could you explain what task your model is trained for and what you want to get with the CAM?

Cheers ✌️

[ryandok]

Hi @frgfm,
Thanks for you help!
My task is semantic segmentation.
And I want to visualize the output of a encoder with the CAM.

[frgfm]

How so?

I mean, semantic segmentation is a task that directly indicates the visual clues that led the model to think that a pixel is class A or B. Do you want to visualize your segmentation output?

I'm not sure I understand the additional information that the CAM would be giving here 🤔

Also, can you share your full code snippet please if you want us to dive into this? :)

[ryandok]

I am so sorry for my late reply!

I am not trying to visualize my segmentation output.

In the segmentation network I designed, there is an attention module in the encoder, and I want to visualize the output of the attention module through the CAM.

Here is my code:
`

def visualization_single_case_batchBased_Nchannel(net, image_list, stride_x, stride_y, stride_z):

  model = SegmentationModelOutputWrapper(net)
  cam_extractor = CAM(model, 'model.attention1')
  w, h, d = image_list[0].shape

  # if the size of image is less than patch_size, then padding it
  add_pad = False
  if w < patch_size[0]:
      w_pad = patch_size[0] - w
      add_pad = True
  else:
      w_pad = 0
  if h < patch_size[1]:
      h_pad = patch_size[1] - h
      add_pad = True
  else:
      h_pad = 0
  if d < patch_size[2]:
      d_pad = patch_size[2] - d
      add_pad = True
  else:
      d_pad = 0
  wl_pad, wr_pad = w_pad // 2, w_pad - w_pad // 2
  hl_pad, hr_pad = h_pad // 2, h_pad - h_pad // 2
  dl_pad, dr_pad = d_pad // 2, d_pad - d_pad // 2
  if add_pad:
      for index, image in enumerate(image_list):
          image_list[index] = np.pad(image, [(wl_pad, wr_pad), (hl_pad, hr_pad), (dl_pad, dr_pad)], mode='constant',
                                     constant_values=0)
  ww, hh, dd = image_list[0].shape

  sx = math.ceil((ww - patch_size[0]) / stride_x) + 1
  sy = math.ceil((hh - patch_size[1]) / stride_y) + 1
  sz = math.ceil((dd - patch_size[2]) / stride_z) + 1
  # print("{}, {}, {}".format(sx, sy, sz))
  cnt = np.zeros(image_list[0].shape).astype(np.float32)
  activation_map = torch.zeros(image_list[0].shape).cuda()

  batch_count = 0
  batch_size = 5
  test_patch_batch = torch.zeros(size=((batch_size, 3,) + patch_size), dtype=torch.float32).cuda()
  total_for_num = sx * sy * sz
  for x in tqdm(range(0, sx)):
      xs = min(stride_x * x, ww - patch_size[0])
      for y in range(0, sy):
          ys = min(stride_y * y, hh - patch_size[1])
          for z in range(0, sz):

              zs = min(stride_z * z, dd - patch_size[2])
              test_patch_list = []
              for image in image_list:
                  test_patch = image[xs:xs + patch_size[0], ys:ys + patch_size[1], zs:zs + patch_size[2]]
                  test_patch_list.append(np.expand_dims(test_patch, axis=0))
              test_patch = np.concatenate(test_patch_list, axis=0).astype(np.float32)
              test_patch = torch.from_numpy(test_patch).cuda()
              test_patch_batch[batch_count] = test_patch
              batch_count += 1
              total_for_num -= 1

              if batch_count == batch_size or total_for_num <= 0:
                  batch_count = 0

                  out = model(test_patch_batch)
                  map = cam_extractor(1, out) # got error here! " File "/home/hra/anaconda3/envs/Pytorch1.7.1/lib/python3.7/site-packages/torchcam/methods/core.py", line 127, in _precheck raise ValueError(f"expected a 1-sized batch to be hooked. Received: {fmap.shape[0]}") ValueError: expected a 1-sized batch to be hooked. Received: 5"
                  map = resize(map, patch_size, order=3)

                  activation_map[xs:xs + patch_size[0], ys:ys + patch_size[1], zs:zs + patch_size[2]] \
                      = activation_map[xs:xs + patch_size[0], ys:ys + patch_size[1], zs:zs + patch_size[2]] + map
                  cnt[xs:xs + patch_size[0], ys:ys + patch_size[1], zs:zs + patch_size[2]] \
                      = cnt[xs:xs + patch_size[0], ys:ys + patch_size[1], zs:zs + patch_size[2]] + 1

  activation_map = activation_map / cnt
  if add_pad:
      activation_map = activation_map[wl_pad:wl_pad + w, hl_pad:hl_pad + h, dl_pad:dl_pad + d]
  return activation_map

`

Best regards,
Ryan

[frgfm]

Hi @ryandok 👋

My apologies, I don't have regular access to github those weeks!

So :

• I understand that you want to check the attention of the encoder of your segmentation model
• you said "the output of the encoder", if that is the case (!= attention), then no need for CAMs. A simple tensor forward hook will work like a charm 👌
• if you indeed mean to get the CAM at this layer, you have to understand that a CAM is a "class" activation map. So this is relative to a class-specific output
• here you model output is not spatially reduced so the CAM doesn't know how to handle it 😅

Now a few suggestion / things to think about :

• you can either visualize the influence of your encoder activation relative to a given class prediction (by backpropagating the spatial sum of your class output)
• or consider the CaM as the spatial influence of this layer a given pixel output prediction.

The first will give you a CAM that looks pretty much like the output segmentation I guess (with torchcam, instead of passing the bare model output, you can pass the spatially reduced version by summation), while the second would be extremely helpful to pinpoint the visual cues that led to the prediction for a specific pixel (here instead of reducing spatially, you'll need to pass the output indexed to select the given pixel) 🙂

Let me know if this isn't clear!

Cheers ✌️

[ryandok]

Hi @frgfm

I am so sorry for my late reply!

Thank you for your advice, which is very helpful to me!

Best regrads,
Ryan