Float Display with fixed precision output uses inconsistent rounding rules

[johnbartholomew]

Rust's f32 formatting seems to follow inconsistent rounding rules when rounding to a fixed output precision.

fn main() {
  for i in 0..=10 {
      let nf = (i as f32) + 0.5f32;
      println!("{:4} -> {:2.0} ; {:5} -> {:3.0}", nf, nf, -nf, -nf);
  }
}

Output (Playground):

 0.5 ->  1 ;  -0.5 ->  -1
 1.5 ->  2 ;  -1.5 ->  -2
 2.5 ->  2 ;  -2.5 ->  -2
 3.5 ->  4 ;  -3.5 ->  -4
 4.5 ->  4 ;  -4.5 ->  -4
 5.5 ->  6 ;  -5.5 ->  -6
 6.5 ->  6 ;  -6.5 ->  -6
 7.5 ->  8 ;  -7.5 ->  -8
 8.5 ->  8 ;  -8.5 ->  -8
 9.5 -> 10 ;  -9.5 -> -10
10.5 -> 10 ; -10.5 -> -10

I get this output on Linux x86-64 with both rustc 1.42.0 (b8cedc0 2020-03-09) and rustc 1.44.0-nightly (f509b26 2020-03-18). I have not tested on other systems, and have not yet tested with f64.

Most inputs are rounded following the rule of rounding half to the nearest integer, but 0.5 and -0.5 are rounded away from zero instead.

When rounding 0.5 to an integer both 0 or 1 are correct outputs, so this may not be a bug. However it seems strange that the round-half-to-even rule should be followed for 1.5, 2.5, and up while a different rule is followed for 0.5.

By comparison, my Python and C versions followed round-half-to-even consistently (rounding 0.5 to 0), though I don't know that they provide any guarantee to follow that rule.

Cross-reference: I originally posted about this on the user forum. Rounding mode for fixed precision float Display format?

[ayushmishra2005]

@johnbartholomew I am not able to find out how Rust is internally handling precision {:2.0}. I check round() of f32. It looks ok.

Can you please guide me?

[ChrisDenton]

I think the formatting code starts at libcore/fmt/float.rs. Which is a wrapper around libcore/num/flt2dec/.

[zyrikby]

I came to this issue also exploring formatting capabilities. For me, it is unclear from the documentation what are the rounding rules. For instance, I cannot explain the following example:

fn main() {
    println!("{:1.3}", 1.2345);    // outputs: 1.234
    println!("{:1.3}", 1.23451);  // outputs: 1.235
}

[johnbartholomew]

@zyrikby That example looks consistent with round-to-even rules.

1.2345 is exactly half way between 1.234 and 1.235, so round-to-even selects 1.234 (ends with an even digit). 1.23451 is closer to 1.235 than it is to 1.234 so it must select 1.235.

Edit: However, the input numbers may not be exactly representable in binary floating point, which could also affect the results. I haven't checked in this case.

[zyrikby]

@johnbartholomew My point here more is that these numbers are not that you would expect to see. For instance, if I implement, e.g., an analogue of Excel I expect to see normal rounding rules applied. However, I am a novice in Rust development, so I do not have a right to impose my opinion to language developers (I do appreciate their work). Still, I think that in the library documentation the fact that round-to-even rules are applied should be mentioned.

[Ixrec]

Floating point rounding rules are one of those things where no matter what you do, it's unintuitive to someone in some cases. Excel has plenty of user confusion from this too, but it's not surprising that Excel chooses round-to-nearest with 5s going up as its default behavior. Programming languages often favor round-to-even because it reduces the biases introduced by rounding every number in a data set, but no Excel user would ever expect that and Excel expressions are more likely to be rounding a single value or only a few values than a large data set.

Regardless, https://doc.rust-lang.org/std/fmt/#precision should definitely specify the rounding behavior. I simply don't know if Rust is doing round-to-even or round-to-zero or something else.

[ajtribick]

This also affects f64:

fn main() {
    for f in -5..5 {
        println!("{0} -> {0:.0}", f as f64 + 0.5f64);
    }
}

gives

-4.5 -> -4
-3.5 -> -4
-2.5 -> -2
-1.5 -> -2
-0.5 -> -1
0.5 -> 1
1.5 -> 2
2.5 -> 2
3.5 -> 4
4.5 -> 4

[Alexendoo]

Triage: Hi, are you still working on this issue @ayushmishra2005?

[ayushmishra2005]

@Alexendoo No, I didn't get chance. Please go head and fix.

[Alexendoo]

@rustbot release-assignment

[dhardy]

Presumably this is also a problem with rounding in the Display impl:

fn main() {
    println!("{}, {}, {}", !0u32, !0u32 as f32, (!0u32 as f32) as u64);
}

4294967295, 4294967300, 4294967296

Why do we see the middle value (2^32 + 4)?

Edit: this is just unnecessary rounding during display (code):

repr: u32=4294967295, f32=4294967300, u64=4294967295
u32→f32→u64 = u32→u64? false
zeros: 0← →0
digits: 32
repr: u32=4294967232, f32=4294967300, u64=4294967232
u32→f32→u64 = u32→u64? false
zeros: 0← →6
digits: 26
repr: u32=4294967168, f32=4294967300, u64=4294967168
u32→f32→u64 = u32→u64? false
zeros: 0← →7
digits: 25
repr: u32=4294967040, f32=4294967000, u64=4294967040
u32→f32→u64 = u32→u64? true
zeros: 0← →8
digits: 24
repr: u32=4294966784, f32=4294966800, u64=4294966784
u32→f32→u64 = u32→u64? true
zeros: 0← →9
digits: 23
repr: u32=4294966272, f32=4294966300, u64=4294966272
u32→f32→u64 = u32→u64? true
zeros: 0← →10
digits: 22

[blueglyph]

I saw the same strange behaviour of Display with f64, and made a comparison between Display::fmt and a trivial rounding function. Display was wrong 50% the time in my small test, and round() was wrong 5% of the time.

The code is here.

Here is the output, with comments where Display is wrong. round() is wrong with 0.135 but otherwise correct.

// "real value": "Display value" <> "round() value" (for positive, then negative values)
0.05 : 0.1  <> 0.1    -0.05:  -0.1  <> -0.1
0.15 : 0.1  <> 0.2    -0.15:  -0.1  <> -0.2
0.25 : 0.2  <> 0.3    -0.25:  -0.2  <> -0.3    // 
0.35 : 0.3  <> 0.4    -0.35:  -0.3  <> -0.4    //
0.45 : 0.5  <> 0.5    -0.45:  -0.5  <> -0.5
0.55 : 0.6  <> 0.6    -0.55:  -0.6  <> -0.6
0.65 : 0.7  <> 0.7    -0.65:  -0.7  <> -0.7
0.75 : 0.8  <> 0.8    -0.75:  -0.8  <> -0.8
0.85 : 0.8  <> 0.9    -0.85:  -0.8  <> -0.9    //
0.95 : 0.9  <> 1.0    -0.95:  -0.9  <> -1.0    //
0.105: 0.10 <> 0.11   -0.105: -0.10 <> -0.11    //
0.115: 0.12 <> 0.12   -0.115: -0.12 <> -0.12
0.125: 0.12 <> 0.13   -0.125: -0.12 <> -0.13    //
0.135: 0.14 <> 0.14   -0.135: -0.14 <> -0.14
0.145: 0.14 <> 0.14   -0.145: -0.14 <> -0.14    //
0.155: 0.15 <> 0.16   -0.155: -0.15 <> -0.16    //
0.165: 0.17 <> 0.17   -0.165: -0.17 <> -0.17
0.175: 0.17 <> 0.18   -0.175: -0.17 <> -0.18    //
0.185: 0.18 <> 0.19   -0.185: -0.18 <> -0.19    //
0.195: 0.20 <> 0.20   -0.195: -0.20 <> -0.20

It's strange that perfectly round values like 0.25 or 0.125, which code well in IEEE-754, are not correctly rounded by Display::fmt of f64. Of course it depends how the decimal conversion vs rounding is done, for example whether it's converted to BCD or not. But the code is complex and I haven't spent enough time to actually find the part responsible for the rounding.

[blueglyph]

Since I already had a routine that rounds a number by processing its string representation, I quickly used it to detect erroneously rounded values in a simple test on more values. Maybe not as good as using a crate like decimal but good enough for a quick investigation.

I found out the ratio of mismatches was 5 / 22 so about 25%, for different number of fractional digits and only testing values ending by 4 or 5, so 0.4, 0.5 using {:.0}, 0.14, 0.15 using {:.1} and so on, since there is little chance to get rounding errors or other digits. That means there is still many numbers in-between that may not round correctly (for ex. 0.4 may round correctly using {:.1}, but not 0.4697872), but it's quicker to quickly scan many values with different precisions.

The ratio remains the same when I check more values by scanning more precisions (same ratio scanning {:.0} - {:.5} and scanning {:.0} - {:.8}).

The code can be found here, if that's any help.

EDIT: added round-to-even check. If I'm only testing 0.*5 values at all depths, about 50% of the values formatted are not correctly rounded, with round-to-even or round-away-from-zero (the rates are pretty close but not identical)

[ajtribick]

I'm not sure that matching the output of round() is the only correct option here. To take an example from C++, std::round is defined to round away from zero (same as Rust's f64::round), while the floating point formatting output depends on the current rounding mode, which on my system defaults to rounding to even.

But I think having a consistent rounding mode in the formatter is a desirable property to have, and it looks like Rust's current situation is mostly round-to-even, regardless of what f64::round does.

[blueglyph]

I'm not sure that matching the output of round() is the only correct option here. To take an example from C++, std::round is defined to round away from zero (same as Rust's f64::round), while the floating point formatting output depends on the current rounding mode, which on my system defaults to rounding to even.

I think I've shown it was not a good solution indeed, since round() doesn't always produce a predictable result. It would need to provide a correct rounding away from zero and rounding to even.