Arrays do not match in python/vm_types_test:test_variant_list_buffers on Windows

[ScottTodd]

Spotted when adding a Windows CI build: https://github.com/iree-org/iree/actions/runs/3414442364/jobs/5682427788, and then reproduced locally.

Full logs:

    Start 1225: iree/runtime/bindings/python/vm_types_test
1225: Test command: C:\Python38\python.exe "D:/dev/projects/iree/runtime/bindings/python/tests/vm_types_test.py"
1225: Working Directory: D:/dev/projects/iree-build/runtime/bindings/python
1225: Environment variables: 
1225:  PYTHONPATH=D:/dev/projects/iree-build/compiler/bindings/python;D:/dev/projects/iree-build/runtime/bindings/python;
1225:  TEST_TMPDIR=D:/dev/projects/iree-build/test_tmpdir/iree/runtime/bindings/python/vm_types_test_test_tmpdir
1225: Test timeout computed to be: 10000000
1225: .INFO:root:variant_list: <VmVariantList(0): []>
1225: .F....
1225: ======================================================================
1225: FAIL: test_variant_list_buffers (__main__.VmTypesTest)
1225: ----------------------------------------------------------------------
1225: Traceback (most recent call last):
1225:   File "D:/dev/projects/iree/runtime/bindings/python/tests/vm_types_test.py", line 70, in test_variant_list_buffers
1225:     np.testing.assert_array_equal(ary1, ary2)
1225:   File "C:\Python38\lib\site-packages\numpy\testing\_private\utils.py", line 934, in assert_array_equal
1225:     assert_array_compare(operator.__eq__, x, y, err_msg=err_msg,
1225:   File "C:\Python38\lib\site-packages\numpy\testing\_private\utils.py", line 844, in assert_array_compare
1225:     raise AssertionError(msg)
1225: AssertionError: 
1225: Arrays are not equal
1225: 
1225: Mismatched elements: 3 / 4 (75%)
1225: Max absolute difference: 4
1225: Max relative difference: 0.5
1225:  x: array([1, 2, 3, 4], dtype=int64)
1225:  y: array([1, 0, 2, 0], dtype=int64)
1225: 
1225: ----------------------------------------------------------------------
1225: Ran 7 tests in 0.005s
1225: 
1225: FAILED (failures=1)
1225: <VmRef vm.list at 0x1a8a7d04ba0>
1225: <VmRef vm.list at 0x1a8a7d04ba0>
1225: <VmVariantList(0): []>
1225: <VmVariantList(0): []>
1225: <VmBuffer size 10 at 0x1a8a7d05030>
4/9 Test #1225: iree/runtime/bindings/python/vm_types_test ........***Failed    0.66 sec

Test code:

iree/runtime/bindings/python/tests/vm_types_test.py

Lines 49 to 72 in 75636a9

	def test_variant_list_buffers(self):
	device = rt.get_device("local-sync")
	ET = rt.HalElementType
	for dt, et in ((np.int8, ET.SINT_8), (np.int16, ET.SINT_16),
	(np.int32, ET.SINT_32), (np.int64, ET.SINT_64),
	(np.uint8, ET.UINT_8), (np.uint16, ET.UINT_16),
	(np.uint32, ET.UINT_32), (np.uint64, ET.UINT_64),
	(np.float32, ET.FLOAT_32), (np.float64, ET.FLOAT_64)):
	# TODO: Unimplemented: (np.float16, ET.FLOAT_16)
	lst = rt.VmVariantList(5)
	ary1 = np.asarray([1, 2, 3, 4], dtype=dt)
	bv1 = device.allocator.allocate_buffer_copy(
	memory_type=rt.MemoryType.DEVICE_LOCAL,
	allowed_usage=(rt.BufferUsage.DEFAULT | rt.BufferUsage.MAPPING),
	buffer=ary1,
	element_type=et)
	lst.push_ref(bv1)
	ary2 = rt.DeviceArray(device,
	lst.get_as_object(0, rt.HalBufferView),
	override_dtype=dt,
	implicit_host_transfer=True)
	np.testing.assert_array_equal(ary1, ary2)
	with self.assertRaises(IndexError):
	lst.get_as_object(1, rt.HalBufferView)

[jpienaar]

This doesn't seem to be failing at head, was this fixed or still an issue?

[ScottTodd]

This doesn't seem to be failing at head, was this fixed or still an issue?

The test is disabled in the script our CI runs:

iree/build_tools/cmake/ctest_all.sh

Lines 87 to 88 in 533c246

	# TODO(#11080): Fix arrays not matching in test_variant_list_buffers
	"iree/runtime/bindings/python/vm_types_test"

I haven't verified if any tests in that list were fixed on their own.

[allieculp]

Hey @ScottTodd checking on this as it's listed as P1. Should we downgrade or add an owner? Or is it fixed?

[ScottTodd]

Likely still an issue, and pretty scary that Python bindings for central functionality are failing some tests on Windows, so P1 is appropriate.

[ScottTodd]

Yep, still an issue. The test_variant_list_buffers test case is failing both the np.int64 and np.uint64 test cases on Windows.

Test code here: https://github.com/openxla/iree/blob/84852a56e051e30960d3facbf050defac06590fd/runtime/bindings/python/tests/vm_types_test.py#L49-L72

[ScottTodd]

Narrowing this down, I'm suspecting pybind behavior is different on Windows (maybe pybind/pybind11#1908 ?).

printf debugging Windows:

1366: dt:  <class 'numpy.int32'>
1366: lst:  <VmVariantList(1): [HalBufferView(4:0x20000011)]>
1366: ary1: [1 2 3 4]
1366: ary2: <IREE DeviceArray: shape=[4], dtype=int32>
1366: dt:  <class 'numpy.int64'>
1366: lst:  <VmVariantList(1): [HalBufferView(4:0x40000011)]>
1366: ary1: [1 2 3 4]
1366: ary2: <IREE DeviceArray: shape=[4], dtype=int32>

and Linux:

1366: dt:  <class 'numpy.int32'>
1366: lst:  <VmVariantList(1): [HalBufferView(4:0x20000011)]>
1366: ary1: [1 2 3 4]
1366: ary2: <IREE DeviceArray: shape=[4], dtype=int32>
1366: dt:  <class 'numpy.int64'>
1366: lst:  <VmVariantList(1): [HalBufferView(4:0x40000011)]>
1366: ary1: [1 2 3 4]
1366: ary2: <IREE DeviceArray: shape=[4], dtype=int64>

(numpy.int64 is mapping to the int32 dtyle and not int64)

[ScottTodd]

I'm testing with and without override_dtype=dt, since [1, 0, 2, 0] suggests some striding issue (I'd expect that to continue as [1, 0, 2, 0, 3, 0, 4, 0] given more space)

[ScottTodd]

Ahhh this is in the code I was just reviewing (#12872)

Per that pybind issue, l is ambiguous (different across Linux/Windows) here? https://github.com/openxla/iree/blob/84852a56e051e30960d3facbf050defac06590fd/runtime/bindings/python/hal.cc#L442-L445