[FLAKY] test_onnx_nodes[llvm-test_nllloss_NCd1d2d3_none_no_weight_negative_ii]

[masahi]

This is causing issues on main:
https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/1630/pipeline
https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/1629/pipeline

It seems this test was introduced in #8872

@AndrewZhaoLuo

[AndrewZhaoLuo]

#8919

[AndrewZhaoLuo]

Thanks for bringing this to my attention, I'll dig a bit deeper into the reasons. But for now to unblock CI^

[AndrewZhaoLuo]

This is quite a weird bug! It seems to result from multiplying elementwise two tensors which do not contain NaN values!

https://github.com/apache/tvm/blob/main/python/tvm/relay/frontend/onnx.py#L3541

This operation seems to be the cause. Interestingly enough loss and mask_tensor dont seem to contain NaNs but after multiplying them the result does! Specifically it occurs where the mask_tensor = 0 only (the intention is to set the indices where the mask_tensor = 0 to 0 in the loss tensor)

[AndrewZhaoLuo]

Even more interesting, the inputs are fixed and not random, so randomness must occur during execution, compilation, etc... The relay graph is fixed so it must be in something with the execution of the graph :0

[AndrewZhaoLuo]

This appears to be because gather does not support negative indices correctly even though it says it does.