-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FLIP proposal to add Access Node event streaming API #73
Conversation
Nice! This is going to be very useful! While the high-level API is described, the low-level details are still unclear to me. Will this API be available for gRPC and the REST API? For gRPC supports streaming, so I assume the endpoints will use that functionality? If the REST API will support event streaming, how will this be implemented? HTTP long polling? Server-Sent Events? |
- `EventType`: Event’s type exactly matches one from the list | ||
- `Contract`: Event was emitted from any contracts from the list | ||
- `Address`: Event was emitted from any contract held by any address from the list |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we also filter events by individual fields in the event object? For example all TokensWithdrawn
events with a specific from
address.
This also helps filtering events by associated dapp in common contracts like NFTStorefront.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I think it should be doable to add second level filtering, but I think there will be some complexity to work out, particularly with arbitrary cadence types in the event payload. I'd prefer to separate this out into its own FLIP. Would you be willing to submit a proposal? Alternatively, we could work together on it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes makes sense. Happy to collaborate on the proposal!
This is great I recently made a poc for this with websocket pubsub in https://events.dnz.dev ( which following state via google bucket ) I have an idea list like below, can give some inspiration. I believe adding a lot of subscription options helps a lit here.
Also I think adding few new AN methods to assist those would be very nice. PS: I am totally going off topic but one more important thing ( which somehow totally ignored till now ) and would be nice to have would have some signature for all these data to prevent rouge AN providing wrong information. |
Another good idea would be numbering events by type, like eventIndex from the creation of the event. That way if I miss some event for some reason ( if I am using this streaming on server side ) I can go and grab missing ones easily. |
Oh also as we are in the subject of |
the examples are based on the PoC gRPC implementation linked at the bottom. This flip is more geared towards alignment on API level. The technical approaches to grpc/rest based streaming are fairly well established.
Initially it would be implemented for gRPC, with REST to follow.
Yes, gRPC will use the native streaming support. See onflow/flow-go#3723 for a PoC
TBD |
Great point. I think once there's an established framework for streaming, it will be straight forward to add for other endpoints/data types.
There is now an endpoint to download There's a proposal in the works to break the |
Do you mean within the stream, or something else?
Can you say more about this? If it's off-topic, we could also discuss in an issue against the flow-go repo |
This post is going to be a loose list of my thoughts around this. Sorry for it not beeing more coherent. This looks like a nice feature however I think it is very important to get this right to avoid confusion. Is the main intended use case here in backend work or will this also be used in UIs? The filtering of events on just event name is too primitive IMHO. Especially since the standard NFT/FT Event patterns allow empty to/from fields. These empty events are very often not needed. An option to filter them away would be good One option here is to allow for more advanced filtering in this api. Another option is to have a separate stream that is an list of events. For instance I want to fetch all Transactions that only withdraw NFT but do not Deposit. If i can use a seperate streaming method on that to group it by transaction or another field. It would also be nice to be able to stream a sealed transaction with result as @bluesign suggests. As well as listen to state changes for a single transaction in an UI. Another use case I can see for this feature is for instance when you open a pack in a UI. You want to see what was revealed to you (if you do packs that way). Having the ability to then subscribe to events of a particular type with a given field filter and group it by transaction would make it very usable. I guess this feature can be composed on top of this api, but an option to actually enrich an Deposit event with data from a script would be nice. For instance add NFTView or something else to Deposit events. |
Yeah I meant individual events level a bit, not sure how it can work but this is the weakest link in security. ( AN - Client communication )
Lets say 'A.0b2a3299cc857e29.TopShot.Deposit' this should have some
basically now state changes seem to be group by Collections ( chunks ), would be nice to get them group by transaction instead. |
This is intended to introduce a mechanism for streaming to the flow Access API, with events being the first usecase. Ideally it will be useful for both front and backend applications. It is OK if it has limited use for frontends in the initial version, as the intention is to iterate.
The vision I'm currently working with is to have a set of core APIs which provide access to all of the data in a convenient way for 80% of usecases. For the remaining 20%, open source modules can be built and shared that provide more advanced features. (80/20 not exact figures) With that said, I think the Access API implementation should include some event payload filtering. I think that should go in a separate FLIP though since there is some additional complexity.
Absolutely. Streaming is planned for more endpoints over time.
Can you say more about grouping by transaction. The event objects returned do include the tx ID. Is that sufficient or is something else needed for this type of usecase?
Yea, this is exactly the type of functionality that could be built into an open sourced module. |
That makes sense. This is definitely something that's needed for a fully trustless Access API. There's still some more work needed on a proof system before this is possible.
We discussed this on discord. To capture the general ask:
I think this is a great idea. This FLIP doesn't include anything about indexing events on nodes. That will eventually be needed so I suggest we include this idea when event indexing is added.
Ah, ok. This would require significant changes to the way state updates are tracked from execution nodes through the execution sync format and clients. I'd be happy to discuss the use cases more if you wanted to open an issue. It's not a quick though. |
I posted a summary of the discussion so far on the forums: We're planning to end the discussion phase for this FLIP on April 14, 2023. |
3723: [Access] Add streaming API for BlockExecutionData r=peterargue a=peterargue Implement streaming gRPC APIs for `BlockExecutionData` and events. Protobuf: onflow/flow#1275 FLIP: onflow/flips#73 Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
3723: [Access] Add streaming API for BlockExecutionData r=peterargue a=peterargue Implement streaming gRPC APIs for `BlockExecutionData` and events. Protobuf: onflow/flow#1275 FLIP: onflow/flips#73 Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
3723: [Access] Add streaming API for BlockExecutionData r=peterargue a=peterargue Implement streaming gRPC APIs for `BlockExecutionData` and events. Protobuf: onflow/flow#1275 FLIP: onflow/flips#73 Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
Thanks for all the input, we're closing this FLIP as accepted. Improvements will be tabled as follow up FLIPs, per the discussion thread |
3723: [Access] Add streaming API for BlockExecutionData r=peterargue a=peterargue Implement streaming gRPC APIs for `BlockExecutionData` and events. Protobuf: onflow/flow#1275 FLIP: onflow/flips#73 Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
3723: [Access] Add streaming API for BlockExecutionData r=peterargue a=peterargue Implement streaming gRPC APIs for `BlockExecutionData` and events. Protobuf: onflow/flow#1275 FLIP: onflow/flips#73 Co-authored-by: Peter Argue <89119817+peterargue@users.noreply.github.com>
Initial draft proposal for the Access Node event streaming API FLIP