Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Event Hubs Client] February Release Documentation #18263

Merged
merged 1 commit into from
Feb 9, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion sdk/eventhub/Azure.Messaging.EventHubs.Processor/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,26 @@
# Release History

## 5.3.0-beta.5 (Unreleased)
## 5.3.0-beta.5 (2021-02-09)

### Changes

#### New Features

- Additional options for tuning load balancing have been added to the `EventProcessorClientOptions`.

- It is now possible to specify a custom endpoint to use for establishing the connection to the Event Hubs service in the `EventHubConnectionOptions` for the processor.

- Interactions with Blob Storage have been tuned for better performance and more efficient resource use. This will also improve start-up time, especially when using the `Greedy` load balancing strategy.

- Errors occurring in the Event Hubs service or active transport are now preserved in full and propagated as an inner exception; this will provide deeper context for diagnosing and troubleshooting exceptions.

- Documentation used for auto-completion via Intellisense and other tools has been enhanced in many areas, addressing gaps and commonly asked questions.

#### Key Bug Fixes

- Upgraded the `Microsoft.Azure.Amqp` library to resolve crashes occurring in .NET 5.

- The calculation for authorization token expiration has been fixed, resulting in fewer token refreshes and network requests.

## 5.3.0-beta.4 (2020-11-10)

Expand All @@ -15,6 +34,8 @@

#### Key Bug Fixes

- Upgraded the `Microsoft.Azure.Amqp` library to resolve crashes occurring in .NET 5.

- The calculation for authorization token expiration has been fixed, resulting in fewer token refreshes and network requests.

## 5.3.0-beta.3 (2020-09-30)
Expand Down
96 changes: 96 additions & 0 deletions ...e.Messaging.EventHubs.Processor/samples/Sample02_EventProcessorConfiguration.md
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,73 @@ The `EventProcessorClient` supports a set of options to configure many aspects o

To begin, please ensure that you're familiar with the items discussed in the [Getting started](https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/eventhub/Azure.Messaging.EventHubs.Processor/samples#getting-started) section of the README, and have the prerequisites and connection string information available.

## Influencing load balancing behavior

To scale event processing, you can run multiple instances of the `EventProcessorClient` and they will coordinate to balance work between them. The responsibility for processing is distributed among each of the active processors configured to read from the same Event Hub and using the same consumer group. To balance work, each active `EventProcessorClient` instance will assume responsibility for processing a set of Event Hub partitions, referred to as "owning" the partitions. The processors collaborate on ownership using storage as a central point of coordination.

While an `EventProcessorClient` is running, it will periodically perform a load balancing cycle in which it audits its own health and inspects the current state of collaboration with other processors. As part of that cycle, it will refresh the timestamp on an ownership record for each partition that it owns. These ownership records help to ensure that each `EventProcessorClient` understands how to maintain its fair share of partitions.

There are several configuration options that can be used together to influence the behavior of load balancing, allowing you to tune it for the specific needs of your application.

### Load balancing strategy

This controls the approach that the `EventProcessorClient` will use to make decisions about how aggressively to request partition ownership; this is most impactful during the initial startup or when recovering from a crash. More information on the strategies available can be found in the [documentation](https://docs.microsoft.com/dotnet/api/azure.messaging.eventhubs.processor.loadbalancingstrategy).

```C# Snippet:EventHubs_Processor_Sample02_LoadBalancingStrategy
var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>";
var blobContainerName = "<< NAME OF THE BLOB CONTAINER >>";

var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>";

var processorOptions = new EventProcessorClientOptions
{
LoadBalancingStrategy = LoadBalancingStrategy.Greedy
};

var storageClient = new BlobContainerClient(
storageConnectionString,
blobContainerName);

var processor = new EventProcessorClient(
storageClient,
consumerGroup,
eventHubsConnectionString,
eventHubName,
processorOptions);
```

### Load balancing intervals

There are two intervals considered during load balancing which can influence its behavior. The `LoadBalancingInterval` controls how frequently a load balancing cycle is run. During the load balancing cycle, the `EventProcessorClient` will attempt to refresh its ownership record for each partition that it owns. The `PartitionOwnershipExpirationInterval` controls how long an ownership record is considered valid. If the processor does not update an ownership record before this interval elapses, the partition represented by this record is considered unowned and is eligible to be claimed by another processor.

```C# Snippet:EventHubs_Processor_Sample02_LoadBalancingIntervals
var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>";
var blobContainerName = "<< NAME OF THE BLOB CONTAINER >>";

var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>";

var processorOptions = new EventProcessorClientOptions
{
LoadBalancingUpdateInterval = TimeSpan.FromSeconds(10),
PartitionOwnershipExpirationInterval = TimeSpan.FromSeconds(30)
};

var storageClient = new BlobContainerClient(
storageConnectionString,
blobContainerName);

var processor = new EventProcessorClient(
storageClient,
consumerGroup,
eventHubsConnectionString,
eventHubName,
processorOptions);
```

## Using web sockets

Communication with the Event Hubs service can be configured by adjusting the `EventHubConfigurationOptions` that are exposed by the `ConnectionOptions` member of a client options type. By default, the `EventProcessorClient` communicates using the AMQP protocol over TCP. Some application host environments prefer to restrict raw TCP socket use, especially in many enterprise or VPN scenarios. In these environments, or when a proxy is in use, communication with the Event Hubs service can make use of web sockets by configuring the client's connection settings.
Expand Down Expand Up @@ -133,6 +200,35 @@ var options = new EventHubConnectionOptions
};
```

### Specifying a custom endpoint address

Connections to the Azure Event Hubs service are made using the fully qualified namespace assigned to the Event Hubs namespace as the connection endpoint address. Because the Event Hubs service uses the endpoint address to locate the corresponding resources, it isn't possible to specify a custom address in the connection string or as the fully qualified namespace.

However, a custom address is required for proper routing by some environments, such as those using unconventional proxy configurations or certain configurations of an Express Route circuit. To support these scenarios, a custom endpoint address may be specified as part of the connection options. This custom address will take precedence for establishing the connection to the Event Hubs service.

```C# Snippet:EventHubs_Processor_Sample02_ConnectionOptionsCustomEndpoint
var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>";
var blobContainerName = "<< NAME OF THE BLOB CONTAINER >>";

var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>";

var processorOptions = new EventProcessorClientOptions();
processorOptions.ConnectionOptions.CustomEndpointAddress = new Uri("amqps://app-gateway.mycompany.com");

var storageClient = new BlobContainerClient(
storageConnectionString,
blobContainerName);

var processor = new EventProcessorClient(
storageClient,
consumerGroup,
eventHubsConnectionString,
eventHubName,
processorOptions);
```

### Configuring the client retry thresholds

The built-in retry policy offers an implementation for an exponential back-off strategy by default, as this provides a good balance between making forward progress and allowing for transient issues that may take some time to resolve. The built-in policy also offers a fixed strategy for those cases where your application requires that you have a deterministic understanding of how long an operation may take.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,13 @@ public TimeSpan LoadBalancingUpdateInterval
///
/// <value>If not specified, an ownership interval of 30 seconds will be assumed.</value>
///
/// <remarks>
/// As a general guideline, it is advised that this value be greater than the configured
/// <see cref="LoadBalancingUpdateInterval" /> by at least a factor of two. It is recommended that
/// this be a factor of three or more, unless there are application scenarios that require more
/// aggressive ownership expiration.
/// </remarks>
///
public TimeSpan PartitionOwnershipExpirationInterval
{
get => _partitionOwnershipExpirationInterval;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using Azure.Messaging.EventHubs.Processor;
using Azure.Messaging.EventHubs.Processor.Tests;
using Azure.Storage.Blobs;
using NUnit.Framework;
Expand All @@ -23,6 +24,91 @@ namespace Azure.Messaging.EventHubs.Tests.Snippets
[SuppressMessage("Style", "IDE0059:Unnecessary assignment of a value", Justification = "Example assignments needed for snippet output content.")]
public class Sample02_EventProcessorConfigurationLiveTests
{
/// <summary>
/// Performs basic smoke test validation of the contained snippet.
/// </summary>
///
[Test]
public void ConfigureLoadBalancingStrategy()
{
#region Snippet:EventHubs_Processor_Sample02_LoadBalancingStrategy

var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>";
var blobContainerName = "<< NAME OF THE BLOB CONTAINER >>";
/*@@*/
/*@@*/ storageConnectionString = StorageTestEnvironment.Instance.StorageConnectionString;
/*@@*/ blobContainerName = "not-real";

var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>";
/*@@*/
/*@@*/ eventHubsConnectionString = EventHubsTestEnvironment.Instance.EventHubsConnectionString;
/*@@*/ eventHubName = "fakeHub";
/*@@*/ consumerGroup = "fakeConsumer";

var processorOptions = new EventProcessorClientOptions
{
LoadBalancingStrategy = LoadBalancingStrategy.Greedy
};

var storageClient = new BlobContainerClient(
storageConnectionString,
blobContainerName);

var processor = new EventProcessorClient(
storageClient,
consumerGroup,
eventHubsConnectionString,
eventHubName,
processorOptions);

#endregion
}

/// <summary>
/// Performs basic smoke test validation of the contained snippet.
/// </summary>
///
[Test]
public void ConfigureLoadBalancingIntervals()
{
#region Snippet:EventHubs_Processor_Sample02_LoadBalancingIntervals

var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>";
var blobContainerName = "<< NAME OF THE BLOB CONTAINER >>";
/*@@*/
/*@@*/ storageConnectionString = StorageTestEnvironment.Instance.StorageConnectionString;
/*@@*/ blobContainerName = "not-real";

var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>";
/*@@*/
/*@@*/ eventHubsConnectionString = EventHubsTestEnvironment.Instance.EventHubsConnectionString;
/*@@*/ eventHubName = "fakeHub";
/*@@*/ consumerGroup = "fakeConsumer";

var processorOptions = new EventProcessorClientOptions
{
LoadBalancingUpdateInterval = TimeSpan.FromSeconds(10),
PartitionOwnershipExpirationInterval = TimeSpan.FromSeconds(30)
};

var storageClient = new BlobContainerClient(
storageConnectionString,
blobContainerName);

var processor = new EventProcessorClient(
storageClient,
consumerGroup,
eventHubsConnectionString,
eventHubName,
processorOptions);

#endregion
}

/// <summary>
/// Performs basic smoke test validation of the contained snippet.
/// </summary>
Expand Down Expand Up @@ -195,6 +281,46 @@ public void ConfigureProxyByProperty()
#endregion
}

/// <summary>
/// Performs basic smoke test validation of the contained snippet.
/// </summary>
///
[Test]
public void ConfigureCustomEndpointAddress()
{
#region Snippet:EventHubs_Processor_Sample02_ConnectionOptionsCustomEndpoint

var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>";
var blobContainerName = "<< NAME OF THE BLOB CONTAINER >>";
/*@@*/
/*@@*/ storageConnectionString = StorageTestEnvironment.Instance.StorageConnectionString;
/*@@*/ blobContainerName = "not-real";

var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>";
/*@@*/
/*@@*/ eventHubsConnectionString = EventHubsTestEnvironment.Instance.EventHubsConnectionString;
/*@@*/ eventHubName = "fakeHub";
/*@@*/ consumerGroup = "fakeConsumer";

var processorOptions = new EventProcessorClientOptions();
processorOptions.ConnectionOptions.CustomEndpointAddress = new Uri("amqps://app-gateway.mycompany.com");

var storageClient = new BlobContainerClient(
storageConnectionString,
blobContainerName);

var processor = new EventProcessorClient(
storageClient,
consumerGroup,
eventHubsConnectionString,
eventHubName,
processorOptions);

#endregion
}

/// <summary>
/// Performs basic smoke test validation of the contained snippet.
/// </summary>
Expand Down
27 changes: 26 additions & 1 deletion sdk/eventhub/Azure.Messaging.EventHubs/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,32 @@
# Release History

## 5.3.0-beta.5 (Unreleased)
## 5.3.0 (2021-02-09)

### Changes

#### New Features

- Connection strings can now be parsed into their key/value pairs using the `EventHubsConnectionStringProperties` class.

- The body of an event has been moved to the `EventData.EventBody` property and makes use of the new `BinaryData` type. To preserve backwards compatibility, the existing `EventData.Body` property has been preserved with the current semantics.

- It is now possible to specify a custom endpoint to use for establishing the connection to the Event Hubs service in the `EventHubConnectionOptions` used by each of the clients.

- Errors occurring in the Event Hubs service or active transport are now preserved in full and propagated as an inner exception; this will provide deeper context for diagnosing and troubleshooting exceptions.

- The `EventHubsModelFactory` has been introduced to provide a single point for creation of Event Hubs model types to assist with mocking and testing.

- Documentation used for auto-completion via Intellisense and other tools has been enhanced in many areas, addressing gaps and commonly asked questions.

#### Key Bug Fixes

- Upgraded the `Microsoft.Azure.Amqp` library to resolve crashes occurring in .NET 5.

- The `EventHubsException.ToString` result will now properly follow the format of other .NET exception output.

- Signaling the cancellation token will no longer cause the `SendAsync` method of the `EventHubProducerClient` to ignore the result of the service operation if publishing has already completed.

- The calculation for authorization token expiration has been fixed, resulting in fewer token refreshes and network requests.

## 5.3.0-beta.4 (2020-11-10)

Expand Down
2 changes: 1 addition & 1 deletion sdk/eventhub/Azure.Messaging.EventHubs/samples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ To quickly create a basic set of Event Hubs resources in Azure and to receive a
Install the Azure Event Hubs client library for .NET with [NuGet](https://www.nuget.org/):

```PowerShell
dotnet add package Azure.Messaging.EventHubs --version 5.3.0-beta.4
dotnet add package Azure.Messaging.EventHubs
```

### Authenticate the client
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,25 @@ var options = new EventHubConnectionOptions
};
```

### Specifying a custom endpoint address

Connections to the Azure Event Hubs service are made using the fully qualified namespace assigned to the Event Hubs namespace as the connection endpoint address. Because the Event Hubs service uses the endpoint address to locate the corresponding resources, it isn't possible to specify another address in the connection string or as the fully qualified namespace.

Some environments using unconventional proxy configurations or with certain configurations of an Express Route circuit require a custom address be used for proper routing, leaving are unable to connect from their on-premises network to the Event Hubs service using the assigned endpoint address. To support these scenarios, a custom endpoint address may be specified as part of the connection options. This custom address will take precedence for establishing the connection to the Event Hubs service.

```C# Snippet:EventHubs_Sample02_ConnectionOptionsCustomEndpoint
var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";

var producerOptions = new EventHubProducerClientOptions();
producerOptions.ConnectionOptions.CustomEndpointAddress = new Uri("amqps://app-gateway.mycompany.com");

var producer = new EventHubProducerClient(
connectionString,
eventHubName,
producerOptions);
```

### Configuring the client retry thresholds

The built-in retry policy offers an implementation for an exponential back-off strategy by default, as this provides a good balance between making forward progress and allowing for transient issues that may take some time to resolve. The built-in policy also offers a fixed strategy for those cases where your application requires that you have a deterministic understanding of how long an operation may take.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<Description>Azure Event Hubs is a highly scalable publish-subscribe service that can ingest millions of events per second and stream them to multiple consumers. This client library allows for both publishing and consuming events using Azure Event Hubs. For more information about Event Hubs, see https://azure.microsoft.com/en-us/services/event-hubs/</Description>
<Version>5.3.0-beta.5</Version>
<Version>5.3.0</Version>
<ApiCompatVersion>5.2.0</ApiCompatVersion>
<PackageTags>Azure;Event Hubs;EventHubs;.NET;AMQP;IoT;$(PackageCommonTags)</PackageTags>
<TargetFrameworks>$(RequiredTargetFrameworks)</TargetFrameworks>
Expand Down
Loading