Add Azure Event Hubs component

[ReubenBond]

We should add an Azure Event Hubs component.
We identified this in the Orleans component PR (#1117), since many Orleans users use Azure Event Hubs for stream processing.

[davidfowl]

I wrote an event hubs sample here (no component but an azure resource):

• https://github.com/davidfowl/AspireEventHub

[oising]

@davidfowl helped me get started this weekend on getting his example integrated into the latest bits, so if you want to assign me, I'm giving it a shot. /cc @ReubenBond

[ReubenBond]

Greatly appreciated, @oising! We're here to help if you run into trouble

[jsquire]

@davidfowl helped me get started this weekend on getting his example integrated into the latest bits, so if you want to assign me, I'm giving it a shot. /cc @ReubenBond

@oising : One thing that I'll mention about David's prototype is that you definitely don't want to use the consumer's ReadEventsAsync for production scenarios unless you've got a very low volume. As noted in the docs, that method is intended to provide an easy "Hello World" experience for exploring Event Hubs. It doesn't provide any fairness guarantees when reading from partitions, does not scale well with a large number of partitions, nor perform under load.

Depending on whether the intention is to allow for multiple instances to be running in the app or a single instance, we'd want to look at PluggableCheckpointStoreEventProcessor<T> or PartitionReceiver.

[oising]

@jsquire This brings up another somewhat orthogonal point. I've yet to read anything about Aspire being aimed at production scenarios (though I can absolutely imagine this, especially with something like Pulumi integrated) -- Aspire finds itself in a weird no-mans land here. It would be good to get messaging around this, because these choices (PartitionReceiver vs EventHubConsumer) are entwined with this vision.

[davidfowl]

The libraries (components) are runtime libraries. We don't have to choose here, we just have a lightweight wrapper around the event hub azure SDK.

[davidfowl]

@sebastienros will help guide this @oising. Feel free to ask questions here!

[jsquire]

The libraries (components) are runtime libraries. We don't have to choose here, we just have a lightweight wrapper around the event hub azure SDK.

Cool. That makes things easier. If the goal is just to wrap/expose clients, then we should consider doing so for:

• EventHubProducerClient
• EventHubProcessorClient
• PartitionReceiver
• EventHubBufferedProducerClient

[jsquire]

@jsquire This brings up another somewhat orthogonal point. I've yet to read anything about Aspire being aimed at production scenarios (though I can absolutely imagine this, especially with something like Pulumi integrated) -- Aspire finds itself in a weird no-mans land here. It would be good to get messaging around this, because these choices (PartitionReceiver vs EventHubConsumer) are entwined with this vision.

That is outside of my purview; I'll defer to @sebastienros and team.

[oising]

The libraries (components) are runtime libraries. We don't have to choose here, we just have a lightweight wrapper around the event hub azure SDK.

Cool. That makes things easier. If the goal is just to wrap/expose clients, then we should consider doing so for:

• EventHubProducerClient
• EventHubProcessorClient
• PartitionReceiver
• EventHubBufferedProducerClient

I don't think there's precedent for exposing multiple clients in this way; even ASB just uses ServiceBusClient I think. I'll play around with some ideas and put up a draft soon.

[jsquire]

I don't think there's precedent for exposing multiple clients in this way; even ASB just uses ServiceBusClient I think. I'll play around with some ideas and put up a draft soon.

Fair enough; if we only get one producer and one consumer, we'd probably want to go with:

• EventHubProducerClient
• EventProcessorClient

[oising]

I don't think there's precedent for exposing multiple clients in this way; even ASB just uses ServiceBusClient I think. I'll play around with some ideas and put up a draft soon.

Fair enough; if we only get one producer and one consumer, we'd probably want to go with:

• EventHubProducerClient
• EventProcessorClient

That's still new territory as far as I can tell. The Aspire component for ASB only has a single top-level/root client ServiceBusClient that needs to be configured. After that, it's vanilla SDK to get a sender or receiver for a queue or a topic. With the Event Hub, there are multiple top-level clients, and some of which have further dependencies, such as a storage account for checkpointing. As far as I understand that how that might work [configuring the storage account], that is vanilla SDK also, even though we may garner the connection reference from an Aspire storage account component. I'm just drinking coffee and figuring it all out. No stress :)

[oising]

I'm thinking of following the pattern in Microsoft.Extensions.Azure:

distributedBuilder.AddEventHubClients(clientBuilder => {
       clientBuilder.AddEventHubProducerClient(...)
       clientBuilder.AddKeyedEventHubProducerClient(...)
       clientBuilder.AddEventProcessorClient(...)
       clientBuilder.AddKeyedEventProcessorClient(...)
       // etc..
       
       clientBuilder.UseCredential( ... )
});

Thoughts?

p.s. I kinda like this pattern for all of Aspire components (at least until they are integrated into the SDK if that happens)

i.e.

builder.AddAspireClients(aspire => { 
    aspire.AddServiceBus( ... )
    aspire.AddStorageAccount( ... )
    // etc 
}); 

It fits within the AddAzureClients pattern, and also helps you a bit more to not conflate the components with the vanilla SDK extension methods.

[davidfowl]

@oising no plans to fundamentally change the pattern from a top-level method, especially for the simple case. We can always add more advanced overloads. We just need to nail the 95% scenario. Customers can always use the lower-level APIs.

[davidfowl]

Done!