Consider a right decimals approach for encoding decimal numbers

[robert-zaremba]

Summary

For blockchain use-case we need to do arithmetic operation which is both deterministic and support big numbers. Moreover, since numbers are used very often we should have an efficient (storage and cpu wise) serialization.

Context

In Cosmos-SDK we are using sdk.Int which is a wrapper around Go big.Int. Big integers are pretty efficient for arithmetic operations. Moreover sdk.Int supports efficient serialization using Int.Bytes()

The problem with sdk.Int is that it's not very friendly to encode Decimal numbers. In the real world we like to think with divisible units, eg: 1.2 USD rather than 120 US Cents

Related Work

We started with an experiment in x/ecocredits to use apd. Currently it's used only in that module.

Considerations

With respect to cosmos/cosmos-sdk#7113, let's consider and be consistent how we want to represent decimal values across Regen Ledger and SDK.

1. Add Bytes serialization for apd
2. Don't use decimal library and use sdk.Int and establish a standard in the SDK community about Decimal numbers representation.

Goal: we should keep in mind, that we don't want to end up with 15 standards of coin / credit / token amount representation.

References:

• Decide about future of sdk.Dec cosmos/cosmos-sdk#7773 : sdk.Dec doesn't work (has bugs).
• Custom decimal places for specific calculations cosmos/cosmos-sdk#4399 Custom decimal places for specific calculations

[aaronc]

I would just note that I'm not sure binary representation is a problem. The SDK has decided to use strings instead. We should discuss there. If we do use binary, I suggest decimal128

[aaronc]

However decimal128 is always 16 bytes. Not so space efficient. An efficient representation would be varints for mantissa and exponent. But again I'm not really sold on needing binary over strings.

[robert-zaremba]

In ecocredtis we are serializing numbers using a decimal string. We can keep adp and use big.Int.Bytes() for serialization in kvstore.

In x/bank we are using a binary serialization:

bz := k.cdc.MustMarshalBinaryBare(&balance)
accountStore.Set([]byte(balance.Denom), bz)

[aaronc]

Which is a proto wrapper around a string not binary. Pretty unnecessary imho to wrap it.

There are reasons we use strings in the SDK. Most notably because it's a standardized representation.

[aaronc]

How and why would we use bigint serialization for decimals? Do you mean mantissa and exponent. Wouldn't a varint be better than big median anyway?

[robert-zaremba]

Mantissa only, because the exponent is int32. We can use varint as well, but I think its an overhead. The serialization is super simple:

<mentisa><exponent>

• mentisa is a low endian int32
• exponent big.Int.Bytes

Deserializing is simple:

• read first 4 bytes and set it to int32
• pass the rest (data[4:]) to Int.SetBytes

The Advantage of this is that we can push this feature to the upstream (apd).

[aaronc]

Seems like varint will save a few bytes right?

[robert-zaremba]

I don't think so. Because we need to encode the mentisa anyway and varint has the bytes length prefix encoded in a sequence of bits (using unary encoding)

[robert-zaremba]

If we will like to optimize a storage, we just encode big.Int and read decimals from other place (eg: BatchInfo).
Simpler optimization is to use only one byte for int32 (exponent) - I can't think about any use case where we will need more than 32 decimal places.

[robert-zaremba]

and apd is limiting exponent as well: apd.MaxExponent = 100000

[aaronc]

Okay well these are details we can optimize later. I first would want to see if we can 1) get buy-in on the SDK side to adopt GDA (cosmos/cosmos-sdk#7773) and 2) whether they're open to a binary format. I personally don't believe we should just invent our own decimal standard which is what was done with sdk.Dec. Do you agree GDA (of some precision) is where we should be starting? Wonder what your thoughts are on Dev's comments in that thread.

[aaronc]

IMHO even int16 should be plenty for exponent no?

[robert-zaremba]

To be honest, I believe that 16 decimal places (int8) is more than enough. This is only for storing. For intermediate operations we could have more precision.

[robert-zaremba]

• unless we want to support negative numbers, then 8 decimals might be not enough.

[ruhatch]

@clevinson can we close this as I think it's meaningfully tracked elsewhere?