Flaky test: test_higher_order_grad.test_arctan and test_higher_order_grad.test_arccosh

[haojin2]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-16728/2/pipeline

======================================================================

FAIL: test_higher_order_grad.test_arctan

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 150, in test_arctan

    check_second_order_unary(array, arctan, grad_grad_op)

  File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 395, in check_second_order_unary

    check_nth_order_unary(x, op, grad_grad_op, 2, rtol, atol)

  File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 465, in check_nth_order_unary

    expected_grad, computed_grad.asnumpy(), rtol=rtol, atol=atol)

  File "/work/mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error nan exceeds tolerance rtol=1.000000e-05, atol=1.000000e-20 (mismatch 0.037037%).

Location of maximum error: (4, 4, 3, 4), a=-0.00000000, b=nan

 ACTUAL: array([[[[-0.        ,  0.        , -0.00001811,  0.        ,

           0.        ],

         [-0.        , -0.        ,  0.        , -0.        ,...

 DESIRED: array([[[[-0.        ,  0.        , -0.00001811,  0.        ,

           0.        ],

         [-0.        , -0.        ,  0.        , -0.        ,...

-------------------- >> begin captured stdout << ---------------------



*** Maximum errors for vector of size 2700:  rtol=1e-05, atol=1e-20



  1: Error nan  Location of error: (4, 4, 3, 4), a=-0.00000000, b=nan



--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1870133712 to reproduce.

--------------------- >> end captured logging << ---------------------

[leezu]

This also affects test_higher_order_grad.test_arccosh. I suppose they are both related, thus adding to this issue.

======================================================================

FAIL: test_higher_order_grad.test_arccosh

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 184, in test_arccosh

    check_second_order_unary(array, arccosh, grad_grad_op)

  File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 395, in check_second_order_unary

    check_nth_order_unary(x, op, grad_grad_op, 2, rtol, atol)

  File "/work/mxnet/tests/python/unittest/test_higher_order_grad.py", line 465, in check_nth_order_unary

    expected_grad, computed_grad.asnumpy(), rtol=rtol, atol=atol)

  File "/work/mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error nan exceeds tolerance rtol=1.000000e-05, atol=1.000000e-20 (mismatch 0.020576%).

Location of maximum error: (0, 3, 6, 5), a=-0.00000000, b=nan

 ACTUAL: array([[[[-0.        , -0.00000031, -0.0000006 , ..., -0.00000129,

           0.00000132, -0.00000008],

         [-0.0000035 , -0.00000132, -0.0000004 , ..., -0.0000015 ,...

 DESIRED: array([[[[-0.        , -0.00000031, -0.0000006 , ..., -0.00000129,

           0.00000132, -0.00000008],

         [-0.0000035 , -0.00000132, -0.0000004 , ..., -0.0000015 ,...

-------------------- >> begin captured stdout << ---------------------


*** Maximum errors for vector of size 4860:  rtol=1e-05, atol=1e-20


  1: Error nan  Location of error: (0, 3, 6, 5), a=-0.00000000, b=nan

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-16810/5/pipeline

[ChaiBapchya]

Also affects test_higher_order_grad.test_arccos

FAIL: test_higher_order_grad.test_arccos
[2020-02-03T02:56:05.913Z] ----------------------------------------------------------------------
[2020-02-03T02:56:05.913Z] Traceback (most recent call last):
[2020-02-03T02:56:05.913Z]   File "C:\Python37\lib\site-packages\nose\case.py", line 198, in runTest
[2020-02-03T02:56:05.913Z]     self.test(*self.arg)
[2020-02-03T02:56:05.913Z]   File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\common.py", line 215, in test_new
[2020-02-03T02:56:05.913Z]     orig_test(*args, **kwargs)
[2020-02-03T02:56:05.913Z]   File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\test_higher_order_grad.py", line 172, in test_arccos
[2020-02-03T02:56:05.913Z]     check_second_order_unary(array, arccos, grad_grad_op)
[2020-02-03T02:56:05.913Z]   File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\test_higher_order_grad.py", line 507, in check_second_order_unary
[2020-02-03T02:56:05.913Z]     check_nth_order_unary(x, op, grad_grad_op, 2, rtol, atol)
[2020-02-03T02:56:05.913Z]   File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\unittest\test_higher_order_grad.py", line 577, in check_nth_order_unary
[2020-02-03T02:56:05.913Z]     expected_grad, computed_grad.asnumpy(), rtol=rtol, atol=atol)
[2020-02-03T02:56:05.913Z]   File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\test_utils.py", line 627, in assert_almost_equal
[2020-02-03T02:56:05.913Z]     raise AssertionError(msg)
[2020-02-03T02:56:05.913Z] AssertionError: 
[2020-02-03T02:56:05.913Z] Items are not equal:
[2020-02-03T02:56:05.913Z] Error nan exceeds tolerance rtol=1.000000e-05, atol=1.000000e-20 (mismatch 0.025510%).
[2020-02-03T02:56:05.913Z] Location of maximum error: (6, 6, 1, 2), a=nan, b=nan
[2020-02-03T02:56:05.913Z]  ACTUAL: array([[[[-2.7580032 ,  0.05628368,  0.03422178, ...,  0.06026911,
[2020-02-03T02:56:05.913Z]            0.07347592,  0.00814795],
[2020-02-03T02:56:05.913Z]          [-0.01800065,  0.0275772 , -0.04048898, ...,  0.06828477,...
[2020-02-03T02:56:05.913Z]  DESIRED: array([[[[-2.7580032 ,  0.05628367,  0.03422177, ...,  0.06026911,
[2020-02-03T02:56:05.913Z]            0.07347594,  0.00814795],
[2020-02-03T02:56:05.913Z]          [-0.01800066,  0.0275772 , -0.04048897, ...,  0.06828477,...
[2020-02-03T02:56:05.913Z] -------------------- >> begin captured stdout << ---------------------
[2020-02-03T02:56:05.913Z] 
[2020-02-03T02:56:05.913Z] *** Maximum errors for vector of size 3920:  rtol=1e-05, atol=1e-20
[2020-02-03T02:56:05.913Z] 
[2020-02-03T02:56:05.913Z]   1: Error nan  Location of error: (6, 6, 1, 2), a=nan, b=nan
[2020-02-03T02:56:05.913Z] 
[2020-02-03T02:56:05.913Z] --------------------- >> end captured stdout << ----------------------
[2020-02-03T02:56:05.913Z] -------------------- >> begin captured logging << --------------------
[2020-02-03T02:56:05.913Z] common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1146143378 to reproduce.
[2020-02-03T02:56:05.913Z] --------------------- >> end captured logging << ---------------------

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17508/1/pipeline/
#17508