Flaky test_random.test_gamma_generator

[marcoabreu]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/398/pipeline

======================================================================

FAIL: test_random.test_gamma_generator

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/workspace/tests/python/unittest/common.py", line 155, in test_new

    orig_test(*args, **kwargs)

  File "/workspace/tests/python/unittest/test_random.py", line 359, in test_gamma_generator

    verify_generator(generator=generator_mx_same_seed, buckets=buckets, probs=probs)

  File "/workspace/python/mxnet/test_utils.py", line 1891, in verify_generator

    str(buckets), str(probs)))

AssertionError: Generator test fails, Chi-square p=[0.48924374659124903, 0.010385813514976699, 0.04690012282396099, 0.024646425434321113, 0.0325781275893177], obs_freq=[array([200090, 200397, 199297, 200077, 200139]), array([199645, 199037, 200196, 201238, 199884]), array([199299, 200081, 201112, 199945, 199563]), array([199821, 201005, 198991, 200373, 199810]), array([200055, 199955, 198868, 200255, 200867])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].

buckets=[(0.0, 0.032092377333650786), (0.032092377333650786, 0.13749794886422795), (0.13749794886422795, 0.3541631504003966), (0.3541631504003966, 0.8211872075748949), (0.8211872075748949, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]

-------------------- >> begin captured stdout << ---------------------

ctx=cpu(0), dtype=float16, Shape=0.5, Scale=1:

ctx=cpu(0), dtype=float16, Shape=1, Scale=5:

ctx=cpu(0), dtype=float32, Shape=0.5, Scale=1:

ctx=cpu(0), dtype=float32, Shape=1, Scale=5:

ctx=cpu(0), dtype=float64, Shape=0.5, Scale=1:



--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1547976354 to reproduce.

--------------------- >> end captured logging << ---------------------

[zheng-da]

Is this problem related to MKLDNN?

[marcoabreu]

No, it's using OpenBLAS.

[zheng-da]

This failure is also caused by floating-point precision and can be reproduced by some specific random seed.

$ export MXNET_TEST_SEED=1547976354
$ nosetests -v tests/python/unittest/test_random.py:test_gamma_generator
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1981327592 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_random.test_gamma_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1547976354 to reproduce.
FAIL

======================================================================
FAIL: test_random.test_gamma_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/ubuntu/incubator-mxnet/tests/python/unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/unittest/test_random.py", line 359, in test_gamma_generator
    verify_generator(generator=generator_mx_same_seed, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.48924374659124903, 0.010385813514976699, 0.046900122823960992, 0.024646425434321113, 0.032578127589317697], obs_freq=[array([200090, 200397, 199297, 200077, 200139]), array([199645, 199037, 200196, 201238, 199884]), array([199299, 200081, 201112, 199945, 199563]), array([199821, 201005, 198991, 200373, 199810]), array([200055, 199955, 198868, 200255, 200867])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.032092377333650786), (0.032092377333650786, 0.13749794886422795), (0.13749794886422795, 0.35416315040039659), (0.35416315040039659, 0.82118720757489494), (0.82118720757489494, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=cpu(0), dtype=float16, Shape=0.5, Scale=1:
ctx=cpu(0), dtype=float16, Shape=1, Scale=5:
ctx=cpu(0), dtype=float32, Shape=0.5, Scale=1:
ctx=cpu(0), dtype=float32, Shape=1, Scale=5:
ctx=cpu(0), dtype=float64, Shape=0.5, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1981327592 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1547976354 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.086s

FAILED (failures=1)

[marcoabreu]

Reopening due to http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1325/pipeline
Python3: GPU:

======================================================================

FAIL: test_operator_gpu.test_gamma_generator

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/usr/local/lib/python3.5/dist-packages/nose/util.py", line 620, in newfunc

    return func(*arg, **kw)

  File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 172, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/gpu/../unittest/test_random.py", line 462, in test_gamma_generator

    verify_generator(generator=generator_mx_same_seed, buckets=buckets, probs=probs)

  File "/work/mxnet/python/mxnet/test_utils.py", line 1950, in verify_generator

    str(buckets), str(probs)))

AssertionError: Generator test fails, Chi-square p=[0.008215479449515959, 0.04392772546045748, 0.023798417315205753, 0.049884893696157506, 0.02361259678655278], obs_freq=[array([199492, 199184, 200223, 201314, 199787]), array([200380, 200923, 199095, 199620, 199982]), array([199517, 199404, 200214, 201221, 199644]), array([200728, 200482, 199673, 200124, 198993]), array([200865, 200658, 199126, 199452, 199899])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].

buckets=[(0.0, 1.1157177565710488), (1.1157177565710488, 2.5541281188299534), (2.5541281188299534, 4.5814536593707755), (4.5814536593707755, 8.047189562170503), (8.047189562170503, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]

-------------------- >> begin captured stdout << ---------------------

ctx=gpu(0), dtype=float16, Shape=0.5, Scale=1:

ctx=gpu(0), dtype=float16, Shape=1, Scale=5:



--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1920693905 to reproduce.

--------------------- >> end captured logging << ---------------------

[Ishitori]

I cannot reproduce the problem using same MXNET_TEST_SEED=1547976354 and MXNET_MODULE_SEED=1981327592.

To fix it, I can think of only the same thing as was done before - decrease success rate. I have decreased it to 0.05 for this particular test, leaving the default 0.15 for other tests.

After running the test sequentially for 100 times, the problem doesn't reproduce itself.

PR is here: #12092