[CosmosDB] Trigger doesn't handle transient connection failures on startup

[brettsam]

A customer had a Cosmos Trigger fail at startup with the stack trace below. After that, the trigger never recovered. It required a host restart in order to get the trigger flowing again:

Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException : The listener for function 'Functions.ProactiveQueue' was unable to start. ---> Microsoft.Azure.Documents.DocumentClientException : Service is currently unavailable, please retry after a while. If this problem persists please contact support.
ActivityId: 7c74fbe6-e4c8-459a-9d2d-24a647fb2f34, Microsoft.Azure.Documents.Common/1.20.0.0
   at async Microsoft.Azure.Documents.Client.ClientExtensions.ParseResponseAsync(HttpResponseMessage responseMessage)
   at async Microsoft.Azure.Documents.Routing.GatewayAddressCache.GetMasterAddressesViaGatewayAsync(ResourceType resourceType,String resourceAddress,String entryUrl,Boolean forceRefresh,Boolean useMasterCollectionResolver)
   at async Microsoft.Azure.Documents.Routing.GatewayAddressCache.ResolveMasterAsync(Boolean forceRefresh)
   at async Microsoft.Azure.Documents.Routing.GatewayAddressCache.TryGetAddresses(DocumentServiceRequest request,PartitionKeyRangeIdentity partitionKeyRangeIdentity,ServiceIdentity serviceIdentity,Boolean forceRefreshPartitionAddresses,CancellationToken cancellationToken)
   at async Microsoft.Azure.Documents.AddressResolver.ResolveAddressesAndIdentityAsync(DocumentServiceRequest request,Boolean forceRefreshPartitionAddresses,CancellationToken cancellationToken)
   at async Microsoft.Azure.Documents.AddressResolver.ResolveAsync(DocumentServiceRequest request,Boolean forceRefreshPartitionAddresses,CancellationToken cancellationToken)
   at async Microsoft.Azure.Documents.AddressSelector.ResolveAddressesAsync(DocumentServiceRequest request,Boolean forceAddressRefresh)
   at async Microsoft.Azure.Documents.AddressSelector.ResolveAllUriAsync(DocumentServiceRequest request,Boolean includePrimary,Boolean forceRefresh)
   at async Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity,Boolean includePrimary,Int32 replicaCountToRead,Boolean requiresValidLsn,Boolean useSessionToken,Boolean checkMinLSN,Boolean forceReadAll)
   at async Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicaAsync(DocumentServiceRequest entity,Boolean includePrimary,Int32 replicaCountToRead,Boolean requiresValidLsn,Boolean useSessionToken,Boolean checkMinLSN,Boolean forceReadAll)
   at async Microsoft.Azure.Documents.QuorumReader.ReadQuorumAsync(DocumentServiceRequest entity,Int32 readQuorum,Boolean includePrimary)
   at async Microsoft.Azure.Documents.QuorumReader.ReadStrongAsync(DocumentServiceRequest entity,Int32 readQuorumValue)
   at async Microsoft.Azure.Documents.ReplicatedResourceClient.InvokeAsync(??)
   at async Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass1.<InvokeAsync>b__0(??)
   at async Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClassf`1.<ExecuteAsync>b__d[T,TPolicyArg1](??)
   at async Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetry[T](Func`1 callbackMethod,Func`3 callShouldRetry,CancellationToken cancellationToken,Action`1 preRetryCallback)
   at async Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetry[T](Func`1 callbackMethod,Func`3 callShouldRetry,CancellationToken cancellationToken,Action`1 preRetryCallback)
   at async Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteAsync[T,TPolicyArg1](??)
   at async Microsoft.Azure.Documents.ReplicatedResourceClient.InvokeAsync(??)
   at async Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync(DocumentServiceRequest request,Func`2 prepareRequestAsyncDelegate)
   at async Microsoft.Azure.Documents.Client.DocumentClient.ReadAsync(DocumentServiceRequest request)
   at async Microsoft.Azure.Documents.Client.DocumentClient.ReadDatabasePrivateAsync(String databaseLink,RequestOptions options,IDocumentClientRetryPolicy retryPolicyInstance)
   at async Microsoft.Azure.Documents.BackoffRetryUtility`1.<>c__DisplayClass2.<ExecuteAsync>b__0[T](??)
   at async Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetry[T](Func`1 callbackMethod,Func`3 callShouldRetry,CancellationToken cancellationToken,Action`1 preRetryCallback)
   at async Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetry[T](Func`1 callbackMethod,Func`3 callShouldRetry,CancellationToken cancellationToken,Action`1 preRetryCallback)
   at async Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteAsync[T](??)
   at async Microsoft.Azure.Documents.ChangeFeedProcessor.ChangeFeedEventHost.InitializeAsync()
   at async Microsoft.Azure.Documents.ChangeFeedProcessor.ChangeFeedEventHost.StartAsync()
   at async Microsoft.Azure.Documents.ChangeFeedProcessor.ChangeFeedEventHost.RegisterObserverFactoryAsync(IChangeFeedObserverFactory factory)
   at async Microsoft.Azure.WebJobs.Extensions.DocumentDB.CosmosDBTriggerListener.StartAsync(CancellationToken cancellationToken)
   at async Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(CancellationToken cancellationToken) 

[brettsam]

It actually looks like this is a dupe of Azure/azure-functions-host#2497. Closing this.

[mathewc]

Yes, Azure/azure-functions-host#2497 will address this in the context of Azure Functions. When using this binding in WebJobs, the new AllowPartialHostStartup config option can be used.