feat: Distribute protobuf files through a new crate

[igamigo]

Addresses #388

[igamigo]

This build script might be overkill. It goes through the protobuf files and builds the list automatically.

Initially I thought not all protobuf files would be needed to be distributed, but when prost compiled rpc.proto it also ended up compiling non-related files as well (like smt.proto) so I added this. I might have missed some config to avoid this from happening.

[igamigo]

PR implementing this on the client: 0xPolygonMiden/miden-client#395

[mFragaBA]

LGTM! I was able to build this without default features. I wasn't able to do so for wasm32_unknown_unknown target though, but it shouldn't be relevant since we're only using the crate in build scripts.

I like the idea of exposing the plain protobuf files, but I also think clients will end up copy-pasting the code to generate the files from the .proto files. Couldn't we expose both the proto files as well as the generated files?

Another thing that came up to mind is whether we'd be able to, from miden-node-proto, generate the files without the ApiClient stuff (I think just commenting:

    // Compile the proto file for all servers APIs
   let protos = &[
       proto_dir.join("block_producer.proto"),
       proto_dir.join("store.proto"),
       proto_dir.join("rpc.proto"),
   ];

from proto/build.rs based on a flag might work), and if those files end up being no_std compatible.

We can iterate on the approach though and I think exposing the proto files is something we just want anyways.

[bobbinth]

I like the idea of exposing the plain protobuf files, but I also think clients will end up copy-pasting the code to generate the files from the .proto files. Couldn't we expose both the proto files as well as the generated files?

Wouldn't generated files need to assume the framework used for generation (e.g., tonic or something else)?

[mFragaBA]

I like the idea of exposing the plain protobuf files, but I also think clients will end up copy-pasting the code to generate the files from the .proto files. Couldn't we expose both the proto files as well as the generated files?

Wouldn't generated files need to assume the framework used for generation (e.g., tonic or something else)?

Hmmm, I'm not sure. Looking at the generated files, the only ones that depend on having tonic are rpc, store and block_producer. The other ones only require having prost, which I think is wasm compatible, isn't it?

[igamigo]

I like the idea of exposing the plain protobuf files, but I also think clients will end up copy-pasting the code to generate the files from the .proto files. Couldn't we expose both the proto files as well as the generated files?

Another thing that came up to mind is whether we'd be able to, from miden-node-proto, generate the files without the ApiClient stuff (I think just commenting:

    // Compile the proto file for all servers APIs
   let protos = &[
       proto_dir.join("block_producer.proto"),
       proto_dir.join("store.proto"),
       proto_dir.join("rpc.proto"),
   ];

from proto/build.rs based on a flag might work), and if those files end up being no_std compatible.

I'm not sure that works, I did a quick test and didn't seem to. Additionally I'm not sure how valuable those created types would be without being wired to some sort of an API client (or similar) and as Bobbin mentioned that probably makes or breaks the possibility of consuming the API/crate properly.

I think eventually we could want to make the API client and independent component (both from the node and miden-client; or maybe just keeping it as a separate crate that can be easily added to projects) but as it stands, keeping it in the client can be a good first step.

[bobbinth]

Looks good! Thank you! I left one last comment inline. After this, we can merge.

[bobbinth]

All looks good! Thank you!

[bobbinth]

oh - i forgot: let's also update the changelog in a small follow-up PR.