AppHost terminates after a little while (kubernetes related exception)

[bgelens]

Currently I'm only defining 2 apps, no containers
Using version 8.0.0-preview.4.24156.9

After some time, the apphost terminates and I get the following exception:

fail: Aspire.Hosting.Dashboard.DcpDataSource[0]
      Watch task over kubernetes Container resources terminated
      System.Net.Http.HttpRequestException: Error while copying content to a stream.
       ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at k8s.LineSeparatedHttpContent.PeekableStreamReader.PeekLineAsync()
         at k8s.LineSeparatedHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context)
         at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
         at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc)
         at k8s.Kubernetes.CreateResultAsync[T](HttpRequestMessage httpRequest, HttpResponseMessage httpResponse, Nullable`1 watch, CancellationToken cancellationToken)
         at k8s.AbstractKubernetes.ICustomObjectsOperations_ListClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at k8s.WatcherExt.<>c__DisplayClass1_0`2.<<MakeStreamReaderCreator>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at k8s.Watcher`1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task`1 t)
         at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
      --- End of stack trace from previous location ---
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
      --- End of stack trace from previous location ---
         at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()
         at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 162
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 162
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Aspire.Hosting.Dashboard.DcpDataSource.<>c__DisplayClass9_1.<<-ctor>g__WatchKubernetesResource|1>d`1.MoveNext() in /_/src/Aspire.Hosting/Dashboard/DcpDataSource.cs:line 70
      --- End of stack trace from previous location ---
         at Aspire.Hosting.Dashboard.DcpDataSource.<>c__DisplayClass9_1.<<-ctor>g__WatchKubernetesResource|1>d`1.MoveNext() in /_/src/Aspire.Hosting/Dashboard/DcpDataSource.cs:line 70

Process finished with exit code 137.

I'm not sure why this happens but maybe it has something to do with unreachable k8s cluster in my current kubectl config?
Is there a way to tell aspire not to do anything with kubernetes?

[davidfowl]

@mitchdenny we need to make sure our retry logic is handling all of the right exception types.

[davidfowl]

This is fixed in p5

[davidfowl]

See #3029