Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

String conversion from decimal (#108) #122

Merged
merged 51 commits into from
Feb 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
bb5f382
relaxed marshaling, slow base 10 parsing
Sep 25, 2022
fb12295
update version
Sep 25, 2022
6866965
add control to benchmark
Sep 25, 2022
55f7ffc
make the ci thing less angry
Sep 25, 2022
5689315
make the ci even less angy
Sep 25, 2022
3c7131f
add fuzz tests
elee1766 Nov 28, 2022
5a106b0
change map
elee1766 Nov 28, 2022
b296df7
magic
elee1766 Nov 28, 2022
cb2c910
clear z on SetFrombig, even if b is nil
elee1766 Nov 28, 2022
df6b926
Update conversion.go
elee1766 Dec 6, 2022
bd117d9
Update conversion.go
Tjudice Dec 14, 2022
048efc0
Merge pull request #1 from Tjudice/patch-1
elee1766 Dec 14, 2022
9a76ab7
base10: minor api change, added docs
holiman Dec 15, 2022
42313ad
ci, go.mod: require go 1.18+
holiman Dec 15, 2022
7e0467a
squashme: circle fixes
holiman Dec 15, 2022
7f61ee9
properly cgheck for overflow in fuzz
elee1766 Dec 15, 2022
1d8f726
number overflow test
elee1766 Dec 15, 2022
f4e6c19
added comments to explain fromBase10Long
elee1766 Dec 15, 2022
84510f7
Update base10.go
elee1766 Dec 15, 2022
6e4f914
Update base10.go
elee1766 Dec 15, 2022
e25ce45
lift up
elee1766 Dec 15, 2022
8420f46
change cutLength to local constant
elee1766 Dec 15, 2022
3fc5f08
better doc
elee1766 Dec 15, 2022
d6f32f6
base10: modify algorithm + fixup tests a bit
holiman Dec 15, 2022
8c858a1
base10: fuzzing found some bugs, which were fixed
holiman Dec 15, 2022
078a8e6
fuzzing: some improvements to the fuzzer
holiman Dec 15, 2022
2413444
add oss-fuzz fuzzer, rename base10 to decimal
holiman Dec 15, 2022
2e6c49f
decimal: fix conversion on base 0, fallback to big.Int
holiman Dec 16, 2022
765b02b
decimal: move some unused constants
holiman Dec 16, 2022
3430ee5
more failing tests
holiman Dec 16, 2022
cbdcdcb
decimal: remove SetString method
holiman Dec 16, 2022
38d53f3
decimal: simplify tests
holiman Dec 16, 2022
ca9a5ea
rm unused constants, fix benchmarks
holiman Dec 16, 2022
9518831
Update decimal.go
elee1766 Dec 17, 2022
ad5ba70
Update conversion.go
elee1766 Dec 17, 2022
301af32
change sql valuer
elee1766 Dec 20, 2022
b5a4906
remove extra struct def
elee1766 Dec 29, 2022
fb7ceef
testing: try to get 100% coverage again
holiman Dec 29, 2022
4ef03d7
lintfix
holiman Dec 29, 2022
5587dd6
fuzzing: fix up fuzzer for string conversion
holiman Dec 29, 2022
7ff0d8f
circle: try to get circleci fuzzing going
holiman Dec 29, 2022
5019647
conversion: more coverage + fix in Scan
holiman Dec 29, 2022
2f0b0b7
decimal_test: more coverage
holiman Dec 29, 2022
dcd75fd
go.mod: fuzzing dep
holiman Dec 29, 2022
0472e9b
conversion: consistent use of ptr receiver
holiman Dec 29, 2022
d55274d
circle: make use of restored corpus
holiman Dec 29, 2022
82fd325
properly parse scientific notation
elee1766 Jan 12, 2023
35d62f2
add some overflow cases for scan, and error on overflow
elee1766 Jan 27, 2023
88226a1
conversion minor nitpicks
holiman Feb 8, 2023
5ff06b0
conversion: fix test
holiman Feb 8, 2023
091c3f9
conversion_test: bring coverage back to 100
holiman Feb 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 12 additions & 46 deletions circle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ commands:

jobs:

go117:
go119:
docker:
- image: cimg/go:1.17
- image: cimg/go:1.19
steps:
- run:
name: "Install tools"
command: |
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.42.0
curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.50.1
- checkout
- run:
name: "Lint"
Expand All @@ -40,18 +40,15 @@ jobs:
command: bash <(curl -s https://codecov.io/bash)
- restore_cache:
keys:
- corpus-v2
- corpus-v3
- run:
name: "Fuzzing"
command: |
go get -u github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build
go-fuzz-build
timeout --preserve-status --signal INT 1m go-fuzz -procs=2
test ! "$(ls crashers)"
GOCACHE=/home/circleci/project/corpus-v3 go test . -run - -fuzz . -fuzztime 1m
- save_cache:
key: corpus-v2-{{ epoch }}
key: corpus-v3-{{ epoch }}
paths:
- corpus
- corpus-v3
- run:
name: "Benchmark"
command: go test -run=- -bench=. -benchmem
Expand All @@ -78,53 +75,22 @@ jobs:
name: "Test (PPC64 emulation)"
command: qemu-ppc64-static uint256.test.ppc64 -test.v

go116:
go118:
docker:
- image: cimg/go:1.16
- image: cimg/go:1.18
steps:
- checkout
- test

go115:
docker:
- image: cimg/go:1.15
steps:
- checkout
- test

go114:
docker:
- image: cimg/go:1.14
steps:
- checkout
- test

go113:
docker:
- image: cimg/go:1.13
steps:
- checkout
- test

go112:
docker:
- image: cimg/go:1.12
steps:
- checkout
- test



workflows:
version: 2
uint256:
jobs:
- go117
- go116
- go115
- go114
- go113
- go112
- go119
- go118
- bigendian:
requires:
- go117
- go119
83 changes: 81 additions & 2 deletions conversion.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,17 @@
package uint256

import (
"database/sql"
"database/sql/driver"
"encoding"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"math/big"
"math/bits"
"strings"
)

const (
Expand All @@ -24,6 +29,16 @@ const (
_ uint = -(maxWords & ^(4 | 8)) // maxWords is 4 or 8.
)

// Compile time interface checks
var (
_ driver.Valuer = (*Int)(nil)
_ sql.Scanner = (*Int)(nil)
_ encoding.TextMarshaler = (*Int)(nil)
_ encoding.TextUnmarshaler = (*Int)(nil)
_ json.Marshaler = (*Int)(nil)
_ json.Unmarshaler = (*Int)(nil)
)

// ToBig returns a big.Int version of z.
func (z *Int) ToBig() *big.Int {
b := new(big.Int)
Expand Down Expand Up @@ -51,12 +66,24 @@ func FromBig(b *big.Int) (*Int, bool) {
return z, overflow
}

// SetFromHex sets z from the given string, interpreted as a hexadecimal number.
// OBS! This method is _not_ strictly identical to the (*big.Int).SetString(..., 16) method.
// Notable differences:
// - This method _require_ "0x" or "0X" prefix.
// - This method does not accept zero-prefixed hex, e.g. "0x0001"
// - This method does not accept underscore input, e.g. "100_000",
// - This method does not accept negative zero as valid, e.g "-0x0",
// - (this method does not accept any negative input as valid)
func (z *Int) SetFromHex(hex string) error {
z.Clear()
return z.fromHex(hex)
}

// fromHex is the internal implementation of parsing a hex-string.
func (z *Int) fromHex(hex string) error {
if err := checkNumberS(hex); err != nil {
return err
}

if len(hex) > 66 {
return ErrBig256Range
}
Expand Down Expand Up @@ -92,6 +119,7 @@ func FromHex(hex string) (*Int, error) {

// UnmarshalText implements encoding.TextUnmarshaler
func (z *Int) UnmarshalText(input []byte) error {
z.Clear()
return z.fromHex(string(input))
}

Expand Down Expand Up @@ -147,7 +175,6 @@ func (z *Int) SetFromBig(b *big.Int) bool {
// specification of minimum digits precision, output field
// width, space or zero padding, and '-' for left or right
// justification.
//
func (z *Int) Format(s fmt.State, ch rune) {
z.ToBig().Format(s, ch)
}
Expand Down Expand Up @@ -470,6 +497,11 @@ func (z *Int) MarshalText() ([]byte, error) {
return []byte(z.Hex()), nil
}

// MarshalJSON implements json.Marshaler.
func (z *Int) MarshalJSON() ([]byte, error) {
return []byte(`"` + z.Hex() + `"`), nil
}

// UnmarshalJSON implements json.Unmarshaler.
func (z *Int) UnmarshalJSON(input []byte) error {
if len(input) < 2 || input[0] != '"' || input[len(input)-1] != '"' {
Expand Down Expand Up @@ -525,6 +557,53 @@ func (z *Int) Hex() string {
return string(output[64-nibbles:])
}

// Scan implements the database/sql Scanner interface.
// It decodes a string, because that is what postgres uses for its numeric type
func (dst *Int) Scan(src interface{}) error {
if src == nil {
dst.Clear()
return nil
}
switch src := src.(type) {
case string:
splt := strings.SplitN(src, "e", 2)
if len(splt) == 1 {
return dst.SetFromDecimal(src)
}
if err := dst.SetFromDecimal(splt[0]); err != nil {
return err
}
if splt[1] == "0" {
return nil
}
Comment on lines +576 to +578
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1e0 should be 1, not 0 ... ?
Not sure what 0e0 is defined as, need to check that.

Copy link
Contributor Author

@elee1766 elee1766 Jan 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't that what this will return? assume X = splt[0], Y = splt[1]

I think XeY woudl be X * 10 ^ Y so 0e0 should be 0 * 10^0 = 0

In the previous lines, i do dst.SetFromDecimal(X)

then if Y == 0, then i should just return, since X*10^0=X

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, yes you're right

exp := new(Int)
if err := exp.SetFromDecimal(splt[1]); err != nil {
return err
}
if !exp.IsUint64() || exp.Uint64() > uint64(len(twoPow256Sub1)) {
return ErrBig256Range
}
exp.Exp(NewInt(10), exp)
_, overflow := dst.MulOverflow(dst, exp)
if overflow {
return ErrBig256Range
}
return nil
case []byte:
return dst.SetFromDecimal(string(src))
}
return fmt.Errorf("cannot scan %T", src)
}

// Value implements the database/sql/driver Valuer interface.
elee1766 marked this conversation as resolved.
Show resolved Hide resolved
// It encodes a base 10 string.
// In Postgres, this will work with both integer and the Numeric/Decimal types
// In MariaDB/MySQL, this will work with the Numeric/Decimal types up to 65 digits, however any more and you should use either VarChar or Char(79)
// In SqLite, use TEXT
func (src *Int) Value() (driver.Value, error) {
return src.ToBig().String(), nil
}

var (
ErrEmptyString = errors.New("empty hex string")
ErrSyntax = errors.New("invalid hex string")
Expand Down
Loading