Event streams

[peterbroadhurst]

Depends on hyperledger/firefly-signer#71 (in a chain with hyperledger/firefly-signer#70)

• Adds internal type event streams
  • These are "upserted" on startup of the block indexer
  • No sophisticated lifecycle management yet
  • Internal event handlers are passed a DB transaction that will be committed with the checkpoint update
  • No other types yet (like gRPC / WebSockets with batch-acknowledgement)
    • Partially because not enough of the engine framework exists
    • Partially because the most critically needed event listener is the state confirmation one
• Component test updated to show its use (fyi @gabriel-indik )
• Builds the event stream system on top of the block indexer
  • Quite a bit different to previous generations of this tech, due to the existence of the index
  • Catchup mode now exploits the DB index of all event signatures, meaning no need to page through blocks
  • No event data is stored in the index, just the events with their signatures
• Optimized for smaller number of high throughput events
  • No selection on address currently (easy to add later if use cases come up)
  • A go channel queue is held for every event stream, buffering events from the main indexer after confirmation there
  • Event streams are passed any logs where topic[0] matches one or more signatures from the ABI
    • Note that ABI signatures are imprecise for parsing (param names, indexed etc.)
  • Detailed parsing of the events happens in the detector for each event stream
  • Matched events are pushed to the dispatcher
  • The dispatcher does batching and delivery to the stream
• The list of events requested in the abi for a stream cannot be modified
  • Deletion and recreation of the stream from scratch would be required
  • Friendly string formatting of events added to firefly-signer in Generate Solidity style full descriptions of functions/events/structs hyperledger/firefly-signer#71 to support this
• Bunch of other framework code in types went in too
  • GORM seems to not support JSON serialization of sub-structs (only sub-arrays) so types.JSONP to support this
  • Added a types.Enum with an updated way to work with enums using generics

[peterbroadhurst]

This is the new feature that gives Solidity-like definitions of events/structs (hopefully useful for various other things too, such as TypedData <-> Struct helpers we've discussed in the past)

[awrichar]

🥳

[awrichar]

what is this sorcery

[awrichar]

TIL "Approximation constraint element" is a thing

[awrichar]

Is it sensible to use gorm First or Take when we're only looking for one record?

[peterbroadhurst]

If you use First (or Take which seems like what we want) it gives you an error, you need to check against:

errors.Is(result.Error, gorm.ErrRecordNotFound)

This seems less readable imho than just using an array and checking in our code.
So I'm leaving it for this PR - as it's what I've done elsewhere.

Would be fine to see a different pattern go in if preferred, but it would I think be worth being standardized in all the places we do this already.