Intermittent NullReferenceException During Send from Azure Functions Host

[jsquire]

Summary

The Azure SDK team received a report of an intermittent issue with the Event Hubs SDK when used with Azure Functions to publish events. It was observed that after a thousand or so SendAsync calls, a NullReferenceException was sometimes observed. The accompanying stack trace indicates that the exception is surfaced by the Microsoft.Azure.Amqp library when sending.

Exception Details

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.Amqp.AmqpLink.SendDelivery(Delivery delivery)
   at Microsoft.Azure.Amqp.AmqpLink.<>c.<Microsoft.Azure.Amqp.IWorkDelegate<Microsoft.Azure.Amqp.Delivery>.Invoke>b__67_0(AmqpLink thisPtr, Delivery paramDelivery, Int32 p1, Int32 p2, Int32 p3)
   at Microsoft.Azure.Amqp.AmqpLink.DoActionIfNotClosed[T1,T2,T3,T4](Func`6 action, T1 p1, T2 p2, T3 p3, T4 p4)
   at Microsoft.Azure.Amqp.AmqpLink.Microsoft.Azure.Amqp.IWorkDelegate<Microsoft.Azure.Amqp.Delivery>.Invoke(Delivery delivery)
   at Microsoft.Azure.Amqp.SerializedWorker`1.DoWorkInternal(T work, Boolean fromList)
   at Microsoft.Azure.Amqp.SerializedWorker`1.DoWork(T work)
   at Microsoft.Azure.Amqp.AmqpLink.StartSendDelivery(Delivery delivery)
   at Microsoft.Azure.Amqp.AmqpLink.TrySendDelivery(Delivery delivery)
   at Microsoft.Azure.Amqp.SendingAmqpLink.Microsoft.Azure.Amqp.IWorkDelegate<Microsoft.Azure.Amqp.AmqpMessage>.Invoke(AmqpMessage message)
   at Microsoft.Azure.Amqp.SerializedWorker`1.DoWorkInternal(T work, Boolean fromList)
   at Microsoft.Azure.Amqp.SerializedWorker`1.DoWork(T work)
   at Microsoft.Azure.Amqp.SendingAmqpLink.SendAsyncResult.Start()
   at Microsoft.Azure.Amqp.WorkCollection`3.StartWork(TKey key, TWork work)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.SendingAmqpLink.EndSendMessage(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Messaging.EventHubs.Amqp.AmqpProducer.SendAsync(Func`1 messageFactory, String partitionKey, CancellationToken cancellationToken)
   at Azure.Messaging.EventHubs.Amqp.AmqpProducer.SendAsync(Func`1 messageFactory, String partitionKey, CancellationToken cancellationToken)
   at Azure.Messaging.EventHubs.Amqp.AmqpProducer.SendAsync(EventDataBatch eventBatch, CancellationToken cancellationToken)
   at Azure.Messaging.EventHubs.Producer.EventHubProducerClient.SendAsync(EventDataBatch eventBatch, CancellationToken cancellationToken)
   at Azure.Messaging.EventHubs.Producer.EventHubProducerClient.SendAsync(EventDataBatch eventBatch, CancellationToken cancellationToken)

Investigation Status

From discussion with @xinchen10, it was believed that the most likely cause was an empty batch of events being sent. I was able to reproduce the exception when doing so in a stress testing environment for the Event Hubs SDK. However, when following-up with @ArmandoLacerda, who originally reported the issue, he feels very certain that there were never empty batches sent in the application scenario.

(The ability to publish an empty batch was an oversight in the client code which has since been patched, but is possible in the current GA version)

Example Snippet

The general scenario that was being observed was building and publishing multiple batches of some number of events within the context of an Azure Function invocation. There are potentially a couple hundred invocations running in parallel - currently each opening their own connection/link to the Event Hubs service, making each invocation self-contained with no shared state. The code to publish is similar to:

await using (var producerClient = new EventHubProducerClient(connectionString, eventHubName);

var batchOptions = new CreateBatchOptions
{
    MaximumSizeInBytes = 800000
};

for (var count = 0; count < 3; ++count)
{
    using var batch = await producerClient.CreateBatchAsync(batchOptions);

    // --
    // Add events to the batch until batch.TryAdd  returns false.
    //--     
      
    await producerClient.SendAsync(eventBatch);
}

References

• [QUESTION] EventBatch doesn't clear after being sent (#16030)
• Event Hubs SDK SendAsync Implementation

[jsquire]

//cc: @xinchen10, @hmlam