hangfire use dryioc report ContainerIsDisposed

[d18zj]

The following exception is reported when a historical job is requeue:

DryIoc.ContainerException: code: Error.ContainerIsDisposed;
message: Container is disposed and should not be used: "container with scope {IsDisposed=true, Name=null}
with Rules with {TrackingDisposableTransients, CaptureContainerDisposeStackTrace} and without {ThrowOnRegisteringDisposableTransient, VariantGenericTypesInResolvedCollection}
with FactorySelector=SelectLastRegisteredFactory
with Made={FactoryMethod=ConstructorWithResolvableArguments} has been DISPOSED!
Dispose stack-trace at DryIoc.Container.Dispose()
at DryIoc.Microsoft.DependencyInjection.DryIocServiceScope.Dispose()
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.Invoke(Object instance, IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.ConfigureBuilder.<>c__DisplayClass4_0.b__0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass15_0.b__1(IApplicationBuilder app)
at Microsoft.AspNetCore.Mvc.Filters.MiddlewareFilterBuilderStartupFilter.<>c__DisplayClass0_0.g__MiddlewareFilterBuilder|0(IApplicationBuilder builder)
at Microsoft.AspNetCore.Server.IIS.Core.IISServerSetupFilter.<>c__DisplayClass2_0.b__0(IApplicationBuilder app)
at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.b__0(IApplicationBuilder app)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at HangfireTest.Program.Main(String[] args)
"
at DryIoc.Throw.It(Int32 error, Object arg0, Object arg1, Object arg2, Object arg3) in //src/DryIoc/Container.cs:line 13766
at DryIoc.Container.ThrowIfContainerDisposed() in //src/DryIoc/Container.cs:line 685
at DryIoc.Container.ResolveAndCache(Int32 serviceTypeHash, Type serviceType, IfUnresolved ifUnresolved) in //src/DryIoc/Container.cs:line 365
at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in //src/DryIoc/Container.cs:line 360
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
at Hangfire.HangfireServiceCollectionExtensions.GetInternalServices(IServiceProvider provider, IBackgroundJobFactory& factory, IBackgroundJobStateChanger& stateChanger, IBackgroundJobPerformer& performer)
at Hangfire.DefaultClientManagerFactory.GetClient(JobStorage storage)
at Hangfire.Dashboard.AspNetCoreDashboardContext.GetBackgroundJobClient()
at Hangfire.Dashboard.RouteCollectionExtensions.<>c__DisplayClass3_0.b__0(DashboardContext context, String jobId)
at Hangfire.Dashboard.BatchCommandDispatcher.Dispatch(DashboardContext context)
at Hangfire.Dashboard.AspNetCoreDashboardMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Program.cs

   public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .UseServiceProviderFactory(new DryIocServiceProviderFactory(new Container().With(rule=>rule.WithCaptureContainerDisposeStackTrace())))
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                });
    }

Startup.cs

public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            var connectionString = Configuration["HangfireOptions:ConnectionString"];
            var hangfireId = int.Parse(Configuration["HangfireOptions:HangfireId"]);

            services.AddHangfire(configuration => configuration
                .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
                .UseSimpleAssemblyNameTypeSerializer()
                .UseRecommendedSerializerSettings()
                .UseRedisStorage(connectionString, new RedisStorageOptions() { Db = hangfireId })
             );

            // Add the processing server as IHostedService
            services.AddHangfireServer();

            services.AddControllersWithViews();
        }

  

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IBackgroundJobClient backgroundJobs, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                app.UseExceptionHandler("/Home/Error");
            }
            app.UseStaticFiles();
            app.UseHangfireDashboard();
            app.UseRouting();

            backgroundJobs.Enqueue(() => Console.WriteLine("Hello world from Hangfire!"));

            app.UseAuthorization();

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllerRoute(
                    name: "default",
                    pattern: "{controller=Home}/{action=Index}/{id?}");
            });
        }
    }

HangfireTest.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="DryIoc.dll" Version="4.8.3" />
    <PackageReference Include="DryIoc.Microsoft.DependencyInjection" Version="5.1.0" />
    <PackageReference Include="Hangfire.AspNetCore" Version="1.7.25" />
    <PackageReference Include="Hangfire.Core" Version="1.7.25" />
    <PackageReference Include="Hangfire.Redis.StackExchange" Version="1.8.5" />
  </ItemGroup>

</Project>

[dadhi]

@d18zj Interesting, thanks for the whole app.
Does it work without the DryIoc?

[d18zj]

@d18zj Interesting, thanks for the whole app. Does it work without the DryIoc?
yes, can work without the DryIoc.

public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                // can work  with comment this line
                .UseServiceProviderFactory(new DryIocServiceProviderFactory(new Container().With(rule=>rule.WithCaptureContainerDisposeStackTrace())))
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                });

You can use this demo to quickly recreate this problem with hangfire's dashboard （dashboard url: http://localhost:5000/hangfire）

[dadhi]

@d18zj Ack. Will look, for now I have no ideas why container is used afterwards. Probably I will search first for the similar errors in other IOCs.

[d18zj]

@dadhi This problem can be replaced with the following test code:

 public class DryIocAdapterSpecificationTests : DependencyInjectionSpecificationTests
    {
        protected override IServiceProvider CreateServiceProvider(IServiceCollection services) => DryIocAdapter.Create(services);

     
        [Fact]
        public void SingletonFactory_Test()
        {
            var collection = new ServiceCollection();
            collection.AddSingleton<ISingletonFactory>(r => new SingletonFactory(r));
            collection.AddTransient<IFakeService, FakeService>();
            var serviceProvider = CreateServiceProvider(collection);
            var scopeFactory = serviceProvider.GetService<IServiceScopeFactory>();
            ISingletonFactory singletonFactory;
            using (var scope = scopeFactory.CreateScope())
            {
                singletonFactory = (ISingletonFactory)scope.ServiceProvider.GetService(typeof(ISingletonFactory));
            }

            var fakeService = singletonFactory.GetService<IFakeService>();
            Assert.NotNull(fakeService);
        }

       
    }

    public interface ISingletonFactory
    {
        T GetService<T>();
    }


    public class SingletonFactory : ISingletonFactory
    {
        private readonly IServiceProvider _serviceProvider;
        public SingletonFactory(IServiceProvider serviceProvider)
        {
            _serviceProvider = serviceProvider;
        }

        public T GetService<T>()
        {
            return (T)_serviceProvider.GetService(typeof(T));
        }
    }

Autofac and Microsoft.Extensions.DependencyInjection can pass the test.

public class SpecificationTests : DependencyInjectionSpecificationTests
    {
        protected override IServiceProvider CreateServiceProvider(IServiceCollection serviceCollection)
        {
            //var builder = new ContainerBuilder();

            //builder.Populate(serviceCollection);

            //IContainer container = builder.Build();
            //return container.Resolve<IServiceProvider>();
            return serviceCollection.BuildServiceProvider();
        }

        [Fact]
        public void SingletonFactory_Test()
        {
            var collection = new ServiceCollection();
            collection.AddSingleton<ISingletonFactory>(r => new SingletonFactory(r));
            collection.AddTransient<IFakeService, FakeService>();
            var serviceProvider = CreateServiceProvider(collection);
            var scopeFactory = serviceProvider.GetService<IServiceScopeFactory>();
            ISingletonFactory singletonFactory;
            using (var scope = scopeFactory.CreateScope())
            {
                singletonFactory = (ISingletonFactory)scope.ServiceProvider.GetService(typeof(ISingletonFactory));
            }

            var fakeService = singletonFactory.GetService<IFakeService>();
            Assert.NotNull(fakeService);
        }

    }

    public interface ISingletonFactory
    {
        T GetService<T>();
    }


    public class SingletonFactory : ISingletonFactory
    {
        private readonly IServiceProvider _serviceProvider;
        public SingletonFactory(IServiceProvider serviceProvider)
        {
            _serviceProvider = serviceProvider;
        }

        public T GetService<T>()
        {
            return (T)_serviceProvider.GetService(typeof(T));
        }
    }

[dadhi]

@d18zj wow, this is much better / simplier thing to check.

[dadhi]

@d18zj Found the bug the injected service provider for the root singletons is the scoped one instead of the root container (which it should be). Fix is in progress

[dadhi]

The DryIoc v4.8.4 with the fix is released

[detoxhby]

@dadhi

Can confirm that this bug still present on v4.8.4.
Last correctly working version is: v4.7.3

Normal .NET 5 (last patch) web app, with builder (where the provider factory uses DryIocAdapter of MS DI under the hood):

.UseServiceProviderFactory(new DryIocServiceProviderFactory(new Container(rules => rules
           .WithoutThrowOnRegisteringDisposableTransient()
           .WithoutThrowIfDependencyHasShorterReuseLifespan()
           .WithCaptureContainerDisposeStackTrace()
           .WithDefaultReuse(Reuse.Scoped)
 , scopeContext: new AsyncLocalScopeContext())))

Dispose exception stack:

ContainerException: code: Error.ContainerIsDisposed;
message: Container is disposed and should not be used: "container with ambient [censored].AsyncLocalScopeContext with scope {Name=null}
with Rules with {TrackingDisposableTransients, CaptureContainerDisposeStackTrace} and without {ThrowIfDependencyHasShorterReuseLifespan, ThrowOnRegisteringDisposableTransient, VariantGenericTypesInResolvedCollection}
with DefaultReuse=Scoped {Lifespan=100}
with FactorySelector=SelectLastRegisteredFactory
with Made={FactoryMethod=ConstructorWithResolvableArguments} has been DISPOSED!
Dispose stack-trace at DryIoc.Container.Dispose()
at DryIoc.Microsoft.DependencyInjection.DryIocServiceScope.Dispose()
at Microsoft.AspNetCore.Http.Features.RequestServicesFeature.DisposeAsync()
at Microsoft.AspNetCore.Http.HttpResponse.<>c.<.cctor>b__36_2(Object disposable)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.FireOnCompleted()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.FireOnCompleted()
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.WriteBody(Boolean flush)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

[dadhi]

@detoxhby Any particular reason to use new AsyncLocalScopeContext()?

[detoxhby]

@dadhi yes, there is a service locator pattern applied and the whole system needs to stay in sync by leveraging same async flow statically

It is somewhat a legacy part of the framework but it cannot be dropped based on usage (I know SL is considered antipattern and I agree in most cases but in huge systems it makes the code more maintainable).

If you have another suggestion to replace this scope context with something different that deals with the same concept of async flow we can adopt that.

[dadhi]

@detoxhby Hello. Given your not as simple setup, I think the fastest way to find the breaking change is to look at the history of changes starting from the working version https://github.com/dadhi/DryIoc/blob/master/docs/DryIoc.Docs/VersionHistory.md
You may help me here at least with the filtering of suspicious commits.

Otherwise (ignoring the luck) I need the smallest repro test for the issue.

Btw, why did you silence the lifespan mismatch check in the rules?

[dadhi]

@detoxhby Anything new on this issue?

[dadhi]

@detoxhby ...and immediately I have found the suspicious change in v4.7.4. Will be checking it.

[detoxhby]

@dadhi yeah we identified the core change but it's not yet trivial for us if that scope/scopecontext handling related changes could be okay with this simple change (e.g. introducing another flag to control this)

mainly, we need the ScopedOrSingleton reuse but with this change the fallback only possible for singleton registrations:
0ef9d64#diff-49030244626c4fbe4b20b50358de6c604be22d0103840811c4753cb50fc4fd35R4823

[dadhi]

seemingly it is fixed

[detoxhby]

@dadhi thanks, gonna test the new release properly next week 👍🏻

[detoxhby]

@dadhi seems like everything is working fine, thanks for rework