@test_approx_eq never finds random Float16 arrays different

[sawcordwell]

When testing approximate equality of Float16 arrays with @test_approx_eq from Base.Test on Julia 0.4 dev, the following produces no error:

using Base.Test
for x = 1:10000
  @test_approx_eq rand(Float16, 10,10) rand(Float16, 10, 10)
end

However, for the Float32 case once should be sufficient to get an assertion error:

@test_approx_eq rand(Float32, 10,10) rand(Float32, 10, 10)

As @johnmyleswhite pointed out when comparing the absolute difference with the eps() of the maximum value, the random arrays are usually different:

rseed(0)
t = 0
for x = 1:10000
  x = rand(Float16, 10, 10)
  y = rand(Float16, 10, 10)
  if all(abs(x - y) .> eps(max(maximum(x), maximum(y))))
    t += 1
  end
end
t / 10000

In this case it is true 90.23 % of the time.

I found that the function in Base.Test to compute the tolerance for @test_approx_eq

array_eps(a) = eps(float(maximum(x->(isfinite(x) ? abs(x) : oftype(x,NaN)), a))) # base/test.jl line 141
va=rand(Float16, 10,10);
vb=rand(Float16, 10,10);
1E4*length(va)*max(array_eps(va), array_eps(vb)) # base/test.jl line 144

will give a value of about 488.

[jiahao]

You've hit upon a corner case for the default heuristic in @test_approx_eq; it simply isn't robust enough for general purpose comparisons. (Related: JuliaLang/LinearAlgebra.jl#67)

The following numerical experiment might help explain what's going on:

julia> x=rand(Float16,10^8);

julia> y=rand(Float16,10^8);

julia> norm(x-y,1) #sum of absolute differences overflows
Inf16

julia> norm(x-y) JuliaLang/julia#2-norm
float16(4082.0)

julia> norm(x-y,Inf) #= max(abs(x-y)), the norm we actually use
float16(1.0)

The l_∞ norm is somewhat too robust to small perturbations, since it retains information only on the maximal difference; your use case is more of "death by a thousand cuts", as it were.

[stevengj]

Given #12472, we now have:

julia> eq = 0; for x = 1:10000
         eq += rand(Float16, 10,10) ≈ rand(Float16, 10, 10)
       end

julia> eq 
0

if we use ≈ (isapprox) rather than @test_approx_eq.

[KristofferC]

@test_approx_eq is now deprecated.