[BUG] pad_sequence with return_mask=True depends on padding_value (incorrect behavior for torch.nan)

[rwicl]

Describe the bug

pad_sequence, called with padding_value=torch.nan and return_mask=True, creates masks in the output tensordict with True-values only, even if there are non-valid entries in the padded tensordict.

To Reproduce

import tensordict
import torch
from tensordict import pad_sequence, TensorDict

tensordict1 = TensorDict({'a': torch.tensor([1., 2., 3.])})
tensordict2 = TensorDict({'a': torch.tensor([4., 5.])})

batch_nan = pad_sequence([tensordict1, tensordict2], padding_value=torch.nan, return_mask=True)
batch_0 = pad_sequence([tensordict1, tensordict2], padding_value=0., return_mask=True)

print(batch_nan['masks']['a'].all())
print(batch_0['masks']['a'].all())

printstensor(True) and tensor(False)

Expected behavior

I expect tensor(False) to be printed in both cases.

[rwicl]

Maybe this behavior makes sense? All entries are valid in the sense that one can read from the nan's their padding-origin directly.

[vmoens]

You're absolutely correct, this is a bug and should be fixed by #1185