Service Bus QuotaExceededException prevents host from starting

[cgillum]

I have a function app with multiple functions. One of them has a Service Bus queue trigger. For some reason calls to the queue fail with QuotaExceededException (details below). This exception in the queue listener is causing the entire host to fail.

Repro steps

Provide the steps required to reproduce the problem

1. Create a function app with a Service Bus trigger.
2. Ensure the Service Bus queue sub/namespace is overquota (not sure how I accomplished this exactly)

Expected behavior

I would expect to be unable to run my Service Bus function. However, I would expect all other trigger types to continue running as they did before.

Actual behavior

Host fails to start and none of my triggers run (including my timer trigger). Here is the relevant exception call stack in the host log file.

2016-10-18T17:10:02.799 A ScriptHost error has occurred
2016-10-18T17:10:02.799 Microsoft.ServiceBus.Messaging.QuotaExceededException: The remote server returned an error: (403) Forbidden. TrackingId:423c1afb-9643-49f8-a368-2fde0436e285,TimeStamp:10/18/2016 5:10:02 PM ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.CreateOrUpdateAsyncResult`1.<GetAsyncSteps>b__1f(CreateOrUpdateAsyncResult`1 thisPtr, IAsyncResult r)
   at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)
   --- End of inner exception stack trace ---
   at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.ServiceBus.NamespaceManager.<CreateQueueAsync>b__8(IAsyncResult r)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.NamespaceManagerExtensions.<CreateQueueIfNotExistsAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusQueueListenerFactory.<CreateAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Listeners.HostListenerFactory.<CreateAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener.<StartAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener.<StartAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.JobHost.<StartAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Script.ScriptHostManager.RunAndBlock(CancellationToken cancellationToken)

Known workarounds

I probably need to fix my Service Bus subscription to get rid of the quota problem.

Related information

Possibly related issue (about listener startup failure behavior): #635

[cgillum]

Update: it turns out I was using the wrong connection string. I specified my Event Hub connection string when I should have been using my Service Bus connection string. Nevertheless, I still expect that this kind of configuration mistake would not prevent the entire host from starting. The error message was a bit misleading, but I do wonder whether a real quota exception might have the same effect of killing the host.

[davidebbo]

It does look like the same root issue as #635.

[mathewc]

Yes, in general any errors happening during host startup or from startup of the various listeners for the functions will prevent the host from starting. This is how the underlying core WebJobs SDK JobHost has always worked. It would require some redesign/rework in the core SDK to change this behavior.

[lindydonna]

@cgillum What was the user error that led to choosing the wrong connection string? We're trying to see how likely it will be that other customers will run into it.

[cgillum]

I was configuring it using app settings and copied the connection string from the wrong resource in the portal. Event Hub and Service Bus connection strings look very similar (same underlying platform and client SDK), so I'm not too surprised I made this mistake.

---

From: Donna Malayeri notifications@github.com
Sent: Monday, October 24, 2016 10:11:07 AM
To: Azure/azure-webjobs-sdk-script
Cc: Chris Gillum; Mention
Subject: Re: [Azure/azure-webjobs-sdk-script] Service Bus QuotaExceededException prevents host from starting (#770)

@cgillumhttps://github.com/cgillum What was the user error that led to choosing the wrong connection string? We're trying to see how likely it will be that other customers will run into it.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/770#issuecomment-255803188, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AClDC3vStMvcZIu6WmI3ddnSSloAkLe1ks5q3OargaJpZM4KaD1F.

[christopheranderson]

Closing since this its not related to host. Following up over email.