Precision issues

[michakraus]

I have numerous problems with precision of ticks in Plots.jl when plotting small numbers. See this issue.

It seems this can be traced back to format_fixed_scientific not working properly. Here is a MWE (tested with Showoff v0.3.1 and Julia v1.3.1):

> format_fixed_scientific(2.99999999999999956E-16, 2, false)
"2.999999999999999556×10⁻¹⁶"

but

> format_fixed_scientific(2.99999999999999999E-16, 2, false)
"3.0×10⁻¹⁶"

[michakraus]

I cannot claim to fully understand the code for format_fixed_scientific, but it seems I could narrow down the problem to originate from the result of

Showoff.jl/src/Showoff.jl

Line 215 in 8502c2e

len, point, neg, digits = grisu((x / 10.0^mag), Base.Grisu.FIXED, grisu_precision)

Consider the following

> x1 = 2.99999999999999956E-16
2.9999999999999994e-16
> x2 = 2.9999999999999999E-16
3.0e-16

I get

> len, point, neg, digits = grisu((x1 / 10.0^mag), Base.Grisu.FIXED, grisu_precision)
(19, 1, false, ...)
> len, point, neg, digits = grisu((x2 / 10.0^mag), Base.Grisu.FIXED, grisu_precision)
(1, 1, false, ...)

with mag=-16 and grisu_precision=18 in both cases, according to the computations in format_fixed_scientific with precision=2. len being 19 in the first case and 1 in the second seems to cause the differences (and troubles).

[michakraus]

I stumbled across

Showoff.jl/src/Showoff.jl

Line 210 in 8502c2e

grisu_precision = precision + abs(round(Int, mag))

Does this actually make sense? Especially in light of x being scaled in

Showoff.jl/src/Showoff.jl

Line 215 in 8502c2e

len, point, neg, digits = grisu((x / 10.0^mag), Base.Grisu.FIXED, grisu_precision)

In my typical examples I have numbers like 6e-15, so that mag=-15. Say we have precision=2 then

grisu_precision = precision + abs(round(Int, mag)) = 17

If we have x = 6.0e-15 then x / 10.0 ^ mag yields 5.999999999999999 and grisu rounds that to 17 digits. If there are no round-off issues in the division of x, grisu will still return only one significant digit, but with round-off errors like this, grisu_precision = 17 causes troubles.

My guess is that setting grisu_precision = precision in all cases solves the problem (it does for me), but I cannot see if there are any side effects to be expected.

[michakraus]

Applying this change leads to two tests failing:

Formatting: Test Failed at /Users/mkraus/DataShare/Julia/Showoff.jl/test/runtests.jl:33
  Expression: Showoff.format_fixed_scientific(0.012345678, 4, true) == "12.34568×10⁻³"
   Evaluated: "12.346×10⁻³" == "12.34568×10⁻³"

Formatting: Test Failed at /Users/mkraus/DataShare/Julia/Showoff.jl/test/runtests.jl:34
  Expression: Showoff.format_fixed_scientific(0.012345678, 4, false) == "1.234568×10⁻²"
   Evaluated: "1.2346×10⁻²" == "1.234568×10⁻²"

Does this make sense? I mean should format_fixed_scientific() when called with precision=4 produce labels with six digits?

[timholy]

Fixed by #19