Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support uint16/32, and tests added #15

Merged
merged 6 commits into from
Jul 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ Opposed to the static `Withdrawal` from the previous section, `ExecutionPayload`
- First up, we will still need to know the static size of the object to avoid costly runtime calculations over and over. Just for reference, that would be the size of all the static fields in the object + 4 bytes for each dynamic field (offset encoding). Feel free to verify the number `512` above.
- If the caller requested only the static size via the `fixed` parameter, return early.
- If the caller, however, requested the total size of the object, we need to iterate over all the dynamic fields and accumulate all their sizes too.
- For all the usual Go suspects like slices and arrays of bytes; 2D sliced and arrays of bytes (i.e. `ExtraData` and `Transactions` above), there are helper methods available in the `ssz` package.
- For all the usual Go suspects like slices and arrays of bytes; 2D sliced and arrays of bytes (i.e. `ExtraData` and `Transactions` above), there are helper methods available in the `ssz` package.
- For types implementing `ssz.StaticObject / ssz.DynamicObject` (e.g. one item of `Withdrawals` above), there are again helper methods available to use them as single objects, static array of objects, of dynamic slice of objects.

The codec itself is very similar to the static example before:
Expand Down Expand Up @@ -228,7 +228,7 @@ For types defined in perfect isolation - dedicated for SSZ - it's easy to define

In reality, often you'll need to encode/decode types which already exist in a codebase, which might not map so cleanly onto the SSZ defined structure spec you want (e.g. you have one union type of `ExecutionPayload` that contains all the Bellatrix, Capella, Deneb, etc fork fields together) and you want to encode/decode them differently based on the context.

Most SSZ libraries will not permit you to do such a thing. Reflection based libraries *cannot* infer the context in which they should switch encoders and can neither can they represent multiple encodings at the same time. Generator based libraries again have no meaningful way to specify optional fields based on different constraints and contexts.
Most SSZ libraries will not permit you to do such a thing. Reflection based libraries *cannot* infer the context in which they should switch encoders and can neither can they represent multiple encodings at the same time. Generator based libraries again have no meaningful way to specify optional fields based on different constraints and contexts.

The only way to handle such scenarios is to write the encoders by hand, and furthermore, encoding might be dependent on what's in the struct, whilst decoding might be dependent on what's it contained within. Completely asymmetric, so our unified *codec definition* approach from the previous sections cannot work.

Expand Down Expand Up @@ -431,7 +431,7 @@ Points of interests to note:

- The generator realized that this type contains dynamic fields (either through `ssz-max` tags or via embedded dynamic objects), so it generated an implementation for `ssz.DynamicObject` (vs. `ssz.StaticObject` in the previous section).
- The generator took into consideration all the size `ssz-size` and `ssz-max` fields to generate serialization calls with different based types and runtime size checks.
- *Note, it is less performant to have runtime size checks like this, so if you know the size of a field, arrays are always preferable vs dynamic lists.*
- *Note, it is less performant to have runtime size checks like this, so if you know the size of a field, arrays are always preferable vs dynamic lists.*

### Cross-validated field sizes

Expand Down Expand Up @@ -551,6 +551,8 @@ The table below is a summary of the methods available for `SizeSSZ` and `DefineS
|:---------------------------:|:---------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------:|
| `bool` | `1 byte` | [`DefineBool`](https://pkg.go.dev/github.com/karalabe/ssz#DefineBool) | [`EncodeBool`](https://pkg.go.dev/github.com/karalabe/ssz#EncodeBool) | [`DecodeBool`](https://pkg.go.dev/github.com/karalabe/ssz#DecodeBool) | [`HashBool`](https://pkg.go.dev/github.com/karalabe/ssz#HashBool) |
| `uint8` | `1 bytes` | [`DefineUint8`](https://pkg.go.dev/github.com/karalabe/ssz#DefineUint8) | [`EncodeUint8`](https://pkg.go.dev/github.com/karalabe/ssz#EncodeUint8) | [`DecodeUint8`](https://pkg.go.dev/github.com/karalabe/ssz#DecodeUint8) | [`HashUint8`](https://pkg.go.dev/github.com/karalabe/ssz#HashUint8) |
| `uint16` | `2 bytes` | [`DefineUint16`](https://pkg.go.dev/github.com/karalabe/ssz#DefineUint16) | [`EncodeUint16`](https://pkg.go.dev/github.com/karalabe/ssz#EncodeUint16) | [`DecodeUint16`](https://pkg.go.dev/github.com/karalabe/ssz#DecodeUint16) | [`HashUint16`](https://pkg.go.dev/github.com/karalabe/ssz#HashUint16) |
| `uint32` | `4 bytes` | [`DefineUint32`](https://pkg.go.dev/github.com/karalabe/ssz#DefineUint32) | [`EncodeUint32`](https://pkg.go.dev/github.com/karalabe/ssz#EncodeUint32) | [`DecodeUint32`](https://pkg.go.dev/github.com/karalabe/ssz#DecodeUint32) | [`HashUint32`](https://pkg.go.dev/github.com/karalabe/ssz#HashUint32) |
| `uint64` | `8 bytes` | [`DefineUint64`](https://pkg.go.dev/github.com/karalabe/ssz#DefineUint64) | [`EncodeUint64`](https://pkg.go.dev/github.com/karalabe/ssz#EncodeUint64) | [`DecodeUint64`](https://pkg.go.dev/github.com/karalabe/ssz#DecodeUint64) | [`HashUint64`](https://pkg.go.dev/github.com/karalabe/ssz#HashUint64) |
| `[N]byte` as `bitvector[N]` | `N bytes` | [`DefineArrayOfBits`](https://pkg.go.dev/github.com/karalabe/ssz#DefineArrayOfBits) | [`EncodeArrayOfBits`](https://pkg.go.dev/github.com/karalabe/ssz#EncodeArrayOfBits) | [`DecodeArrayOfBits`](https://pkg.go.dev/github.com/karalabe/ssz#DecodeArrayOfBits) | [`HashArrayOfBits`](https://pkg.go.dev/github.com/karalabe/ssz#HashArrayOfBits) |
| `bitfield.Bitlist`² | [`SizeSliceOfBits`](https://pkg.go.dev/github.com/karalabe/ssz#SizeSliceOfBits) | [`DefineSliceOfBitsOffset`](https://pkg.go.dev/github.com/karalabe/ssz#DefineSliceOfBitsOffset) [`DefineSliceOfBitsContent`](https://pkg.go.dev/github.com/karalabe/ssz#DefineSliceOfBitsContent) | [`EncodeSliceOfBitsOffset`](https://pkg.go.dev/github.com/karalabe/ssz#EncodeSliceOfBitsOffset) [`EncodeSliceOfBitsContent`](https://pkg.go.dev/github.com/karalabe/ssz#EncodeSliceOfBitsContent) | [`DecodeSliceOfBitsOffset`](https://pkg.go.dev/github.com/karalabe/ssz#DecodeSliceOfBitsOffset) [`DecodeSliceOfBitsContent`](https://pkg.go.dev/github.com/karalabe/ssz#DecodeSliceOfBitsContent) | [`HashSliceOfBits`](https://pkg.go.dev/github.com/karalabe/ssz#HashSliceOfBits) |
Expand Down Expand Up @@ -582,7 +584,7 @@ The package includes a set of benchmarks for handling the beacon spec types and
If you want to see the performance on a more realistic piece of data, you'll need to provide a beacon state SSZ object and place it into the project root named `state.ssz`. You can then run `go test --bench=Mainnet ./tests/manual_test.go` to explicitly run this one benchmark. A sample output running against a 208MB state export from around June 11, 2024, on a MacBook Pro M2 Max:

```
go test --bench=Mainnet ./tests/manual_test.go
go test --bench=Mainnet ./tests/manual_test.go

BenchmarkMainnetState/beacon-state/208757379-bytes/encode-12 26 45164494 ns/op 4622.16 MB/s 74 B/op 0 allocs/op
BenchmarkMainnetState/beacon-state/208757379-bytes/decode-12 27 40984980 ns/op 5093.51 MB/s 8456490 B/op 54910 allocs/op
Expand Down
20 changes: 20 additions & 0 deletions cmd/sszgen/opset.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,26 @@ func (p *parseContext) resolveBasicOpset(typ *types.Basic, tags *sizeTag) (opset
"DecodeUint8({{.Codec}}, &{{.Field}})",
[]int{1},
}, nil
case types.Uint16:
if tags != nil && tags.size[0] != 2 {
return nil, fmt.Errorf("uint16 basic type requires ssz-size=2: have %d", tags.size[0])
}
return &opsetStatic{
"DefineUint16({{.Codec}}, &{{.Field}})",
"EncodeUint16({{.Codec}}, &{{.Field}})",
"DecodeUint16({{.Codec}}, &{{.Field}})",
[]int{2},
}, nil
case types.Uint32:
if tags != nil && tags.size[0] != 4 {
return nil, fmt.Errorf("uint32 basic type requires ssz-size=4: have %d", tags.size[0])
}
return &opsetStatic{
"DefineUint32({{.Codec}}, &{{.Field}})",
"EncodeUint32({{.Codec}}, &{{.Field}})",
"DecodeUint32({{.Codec}}, &{{.Field}})",
[]int{4},
}, nil
case types.Uint64:
if tags != nil && tags.size[0] != 8 {
return nil, fmt.Errorf("uint64 basic type requires ssz-size=8: have %d", tags.size[0])
Expand Down
26 changes: 26 additions & 0 deletions codec.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,32 @@ func DefineUint8[T ~uint8](c *Codec, n *T) {
HashUint8(c.has, *n)
}

// DefineUint16 defines the next field as a uint16.
func DefineUint16[T ~uint16](c *Codec, n *T) {
if c.enc != nil {
EncodeUint16(c.enc, *n)
return
}
if c.dec != nil {
DecodeUint16(c.dec, n)
return
}
HashUint16(c.has, *n)
}

// DefineUint32 defines the next field as a uint32.
func DefineUint32[T ~uint32](c *Codec, n *T) {
if c.enc != nil {
EncodeUint32(c.enc, *n)
return
}
if c.dec != nil {
DecodeUint32(c.dec, n)
return
}
HashUint32(c.has, *n)
}

// DefineUint64 defines the next field as a uint64.
func DefineUint64[T ~uint64](c *Codec, n *T) {
if c.enc != nil {
Expand Down
38 changes: 38 additions & 0 deletions decoder.go
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,44 @@ func DecodeUint8[T ~uint8](dec *Decoder, n *T) {
}
}

// DecodeUint16 parses a uint16.
func DecodeUint16[T ~uint16](dec *Decoder, n *T) {
if dec.err != nil {
return
}
if dec.inReader != nil {
_, dec.err = io.ReadFull(dec.inReader, dec.buf[:2])
*n = T(binary.LittleEndian.Uint16(dec.buf[:2]))
dec.inRead += 2
} else {
if len(dec.inBuffer) < 2 {
dec.err = io.ErrUnexpectedEOF
return
}
*n = T(binary.LittleEndian.Uint16(dec.inBuffer))
dec.inBuffer = dec.inBuffer[2:]
}
}

// DecodeUint32 parses a uint32.
func DecodeUint32[T ~uint32](dec *Decoder, n *T) {
if dec.err != nil {
return
}
if dec.inReader != nil {
_, dec.err = io.ReadFull(dec.inReader, dec.buf[:4])
*n = T(binary.LittleEndian.Uint32(dec.buf[:4]))
dec.inRead += 4
} else {
if len(dec.inBuffer) < 4 {
dec.err = io.ErrUnexpectedEOF
return
}
*n = T(binary.LittleEndian.Uint32(dec.inBuffer))
dec.inBuffer = dec.inBuffer[4:]
}
}

// DecodeUint64 parses a uint64.
func DecodeUint64[T ~uint64](dec *Decoder, n *T) {
if dec.err != nil {
Expand Down
28 changes: 28 additions & 0 deletions encoder.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,34 @@ func EncodeUint8[T ~uint8](enc *Encoder, n T) {
}
}

// EncodeUint16 serializes a uint16.
func EncodeUint16[T ~uint16](enc *Encoder, n T) {
if enc.outWriter != nil {
if enc.err != nil {
return
}
binary.LittleEndian.PutUint16(enc.buf[:2], (uint16)(n))
_, enc.err = enc.outWriter.Write(enc.buf[:2])
} else {
binary.LittleEndian.PutUint16(enc.outBuffer, (uint16)(n))
enc.outBuffer = enc.outBuffer[2:]
}
}

// EncodeUint32 serializes a uint32.
func EncodeUint32[T ~uint32](enc *Encoder, n T) {
if enc.outWriter != nil {
if enc.err != nil {
return
}
binary.LittleEndian.PutUint32(enc.buf[:4], (uint32)(n))
_, enc.err = enc.outWriter.Write(enc.buf[:4])
} else {
binary.LittleEndian.PutUint32(enc.outBuffer, (uint32)(n))
enc.outBuffer = enc.outBuffer[4:]
}
}

// EncodeUint64 serializes a uint64.
func EncodeUint64[T ~uint64](enc *Encoder, n T) {
// Nope, dive into actual encoding
Expand Down
14 changes: 14 additions & 0 deletions hasher.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,20 @@ func HashUint8[T ~uint8](h *Hasher, n T) {
h.insertChunk(buffer, 0)
}

// HashUint16 hashes a uint16.
func HashUint16[T ~uint16](h *Hasher, n T) {
var buffer [32]byte
binary.LittleEndian.PutUint16(buffer[:], uint16(n))
h.insertChunk(buffer, 0)
}

// HashUint32 hashes a uint32.
func HashUint32[T ~uint32](h *Hasher, n T) {
var buffer [32]byte
binary.LittleEndian.PutUint32(buffer[:], uint32(n))
h.insertChunk(buffer, 0)
}

// HashUint64 hashes a uint64.
func HashUint64[T ~uint64](h *Hasher, n T) {
var buffer [32]byte
Expand Down
2 changes: 2 additions & 0 deletions tests/consensus_specs_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ func commonPrefix(a []byte, b []byte) []byte {
// consensus spec tests repo and runs the encoding/decoding/hashing round.
func TestConsensusSpecBasics(t *testing.T) {
testConsensusSpecBasicType[*types.SingleFieldTestStruct](t, "SingleFieldTestStruct")
testConsensusSpecBasicType[*types.SmallTestStruct](t, "SmallTestStruct")
testConsensusSpecBasicType[*types.FixedTestStruct](t, "FixedTestStruct")
testConsensusSpecBasicType[*types.BitsStruct](t, "BitsStruct")
}

Expand Down
17 changes: 17 additions & 0 deletions tests/testtypes/consensus-spec-tests/gen_fixed_test_struct_ssz.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 16 additions & 0 deletions tests/testtypes/consensus-spec-tests/gen_small_test_struct_ssz.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions tests/testtypes/consensus-spec-tests/types_basics.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,25 @@ package consensus_spec_tests
import "github.com/prysmaticlabs/go-bitfield"

//go:generate go run -cover ../../../cmd/sszgen -type SingleFieldTestStruct -out gen_single_field_test_struct_ssz.go
//go:generate go run -cover ../../../cmd/sszgen -type SmallTestStruct -out gen_small_test_struct_ssz.go
//go:generate go run -cover ../../../cmd/sszgen -type FixedTestStruct -out gen_fixed_test_struct_ssz.go
karalabe marked this conversation as resolved.
Show resolved Hide resolved
//go:generate go run -cover ../../../cmd/sszgen -type BitsStruct -out gen_bits_struct_ssz.go

type SingleFieldTestStruct struct {
A byte
}

type SmallTestStruct struct {
A uint16
B uint16
}

type FixedTestStruct struct {
A uint8
B uint64
C uint32
}

type BitsStruct struct {
A bitfield.Bitlist `ssz-max:"5"`
B [1]byte `ssz-size:"2" ssz:"bits"`
Expand Down