DynamicPropertyRegistry Values Not Set After Dev Tools Reload for RestartScope Containers

[shawnweeks]

When using a combination of @RestartScope Test Containers and the DynamicPropertyRegistry registry property values are not set back after Spring Dev Tools reload the project. The issue was partially resolved in #35786 but that fix does not appear to resolve the entire issue. I've linked to a small example project demonstrating the issue. See https://github.com/shawnweeks/spring_boot_testcontainers_restartscope_issue

[nosan]

I did some research regarding this bug, and frankly, it is quite a tricky one.

First run:

TestcontainersLifecycleBeanPostProcessor
  -- initializeContainers()

RestartScopeInitializer.RestartScope
  -- Restarter.getOrAddAttribute('redisContainer', ObjectFactory<?> factory)
  -- 'redisContainer' does not exist in attributes, so ObjectFactory<?> is being called.

TestcontainersPropertySourceAutoConfiguration.dynamicPropertyRegistry() is being called because it is needed for 'redisContainer'
-- Attach 'testcontainersPropertySource'
-- Register bean definition named: EventPublisherRegistrar.class.getName()

TestcontainersConfiguration.redisContainer()
  -- create RedisContainer
  -- Set RedisContainer properties to the 'dynamicPropertyRegistry'

TestcontainersLifecycleBeanPostProcessor.initializeStartables()
 -- Start containers

DemoController @Value annotations are being processed.

Restart:

TestcontainersLifecycleBeanPostProcessor
  -- initializeContainers()
RestartScopeInitializer.RestartScope
  -- Restarter.getOrAddAttribute("redisContainer", ObjectFactory<?>)
  -- "redisContainer" exists in attributes, so ObjectFactory<?> **is not being called.**

DemoController @Value annotations are not being processed due to Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'MY_REDIS_HOST' in value "${MY_REDIS_HOST}"

There are two potential issues with this bug.

The first one TestcontainersPropertySourceAutoConfiguration.dynamicPropertyRegistry()
is being called too late if there is no direct reference to it, so TestcontainersPropertySource will not be available during @Value annotation processing.

The second one is that methods annotated @RestartScope are being called only once. That is why dynamicPropertyRegistry will not be created earlier, and container properties are ignored. (Properties are being set inside the method).

A potential fix is:

• If properties come from @RestartScope methods they should be shared across restarts.
• TestcontainersPropertySource should be registered earlier.

This branch contains an implementation of the fix that I have tried to describe above.
main...nosan:spring-boot:41552

I used this example for research: https://github.com/shawnweeks/spring_boot_testcontainers_restartscope_issue

[philwebb]

Thanks for the detailed analysis @nosan. This is indeed a complicated issue to fix and made even more difficult by the changes we're introducing in 3.4 due to #41996.

I'm not really sure if we should attempt to fix this in 3.3 since I don't really like the fact that TestcontainersPropertySource needs to know about restart scope.

The following code does appear to work with 3.4.0-SNAPSHOT:

@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {

    @Bean
    @RestartScope
    public GenericContainer<?> redisContainer() {
        GenericContainer<?> redisContainer = new GenericContainer<>("redis:7");
        redisContainer.withExposedPorts(6379);
        redisContainer.withCommand("redis-server", "--requirepass redis_user", "--save 60 1", "--loglevel debug");
        return redisContainer;
    }

    @Bean
    public DynamicPropertyRegistrar redisContainerProperties(GenericContainer<?> redisContainer) {
    	return (registry) -> {
            registry.add("MY_REDIS_HOST", () -> "localhost");
            registry.add("MY_REDIS_PORT", () -> redisContainer.getMappedPort(6379));
            registry.add("MY_REDIS_PASSWORD", () -> "redis_user");
    	};
    }

}

This split allows the container to remain in the @RestartScope whilst the properties are always recreated.

[philwebb]

The following ugly hack will work with 3.3.x:

package com.example.demo;

import java.util.List;

import org.springframework.boot.devtools.restart.RestartScope;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.testcontainers.containers.ContainerState;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.lifecycle.Startable;

import com.github.dockerjava.api.command.InspectContainerResponse;

@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {

	@Bean
	@RestartScope
	public GenericContainer<?> redisContainer(DynamicPropertyRegistry registry) {
		GenericContainer<?> redisContainer = new GenericContainer<>("redis:7");
		redisContainer.withExposedPorts(6379);
		redisContainer.withCommand("redis-server", "--requirepass redis_user", "--save 60 1", "--loglevel debug");
		return redisContainer;
	}

	@Bean
	public RedisProperties redisProperties(GenericContainer<?> redisContainer, DynamicPropertyRegistry registry) {
		redisContainer.start();
		registry.add("MY_REDIS_HOST", () -> "localhost");
		registry.add("MY_REDIS_PORT", () -> redisContainer.getMappedPort(6379));
		registry.add("MY_REDIS_PASSWORD", () -> "redis_user");
		return new RedisProperties();
	}

	static class RedisProperties implements Startable, ContainerState {

		@Override
		public void start() {
		}

		@Override
		public void stop() {
		}

		@Override
		public List<Integer> getExposedPorts() {
			return null;
		}

		@Override
		public InspectContainerResponse getContainerInfo() {
			return null;
		}

	}

}

[philwebb]

Flagging to see if the team think we should still attempt a fix in 3.2/3.3.

[nosan]

I propose adding a note to the documentation explaining that DynamicPropertyRegistry does not work with restart-scoped containers (beans), along with the reason behind this limitation.
For version 3.4.x, include a reference to DynamicPropertyRegistrar and mention that it works with restart-scoped containers (beans).

Fixing this issue in versions 3.2/3.3 would result in inconsistency with 3.4.x, as the latter will not receive a similar fix.

[nosan]

This hack works with 3.2.x, 3.3.x and 3.4.0-M3

@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {

	@Bean
	@RestartScope
	public GenericContainer<?> redisContainer() {
		GenericContainer<?> redisContainer = new GenericContainer<>("redis:7");
		redisContainer.withExposedPorts(6379);
		redisContainer.withCommand("redis-server", "--requirepass redis_user", "--save 60 1", "--loglevel debug");
		return redisContainer;
	}

	@Autowired
	void registryRedisContainerProperties(@Lazy GenericContainer<?> redisContainer, DynamicPropertyRegistry registry) {
		registry.add("MY_REDIS_HOST", () -> "localhost");
		registry.add("MY_REDIS_PORT", () -> redisContainer.getMappedPort(6379));
		registry.add("MY_REDIS_PASSWORD", () -> "redis_user");
	}

}

[wilkinsona]

It'll fail by default with 3.4.0-SNAPSHOT due to the deprecation of support for injecting DynamicPropertyRegistry.

[philwebb]

We're going to leave this one open to see if we can fix it in some way, but it's hard to see us getting to it quickly. Upgrading to 3.4. when it's out is probably the best option for anyone with this issue.

[shawnweeks]

I've updated to 3.4.1 and am using the new DynamicPropertyRegistrar however there are still issues namely the DynamicPropertyRegistrar bean is created early in the startup process however the initialize method is called too late in the process for all the placeholders in my data sources to resolve. On 3.3.x I could use a bean post processor on the test side to force a dependency between my data source classes and dynamicPropertyRegistry but this no longer works.

[wilkinsona]

@shawnweeks can you please update https://github.com/shawnweeks/spring_boot_testcontainers_restartscope_issue along those lines so that we can take a look?

[shawnweeks]

I'm probably misunderstanding why the issue happens but here is an example, sorry it's a little convoluted but it's based on a real application with the same issue. See https://github.com/shawnweeks/spring_boot_testcontainers_restartscope_issue/tree/issue_v2

[philwebb]

@shawnweeks I think the issue in the sample is the demoFilter bean is causing early initialization. This is a general problem, not just related to Testcotnainers. There's a small warning in the docs about this.

You can use a DelegatingFilterProxyRegistrationBean to get around the problem:

@Bean
DelegatingFilterProxyRegistrationBean demoFilterRegistration() {
    return new DelegatingFilterProxyRegistrationBean("demoFilter");
}

[shawnweeks]

I've updated the example, in the real program the filters are for Spring Security and it appears that what you suggest works when used by it self but if your using FilterRegistrationBean to disable registration the original issue comes back. The filter causes everything to initialize too early. This issue didn't occur setting everything up the pre 3.4.x way.

[shawnweeks]

If I change to something like this everything appears to work the new way.

    @Bean
    public DelegatingFilterProxyRegistrationBean demoFilterProxyRegistration() {
        DelegatingFilterProxyRegistrationBean delegatingFilterProxyRegistrationBean = new DelegatingFilterProxyRegistrationBean("demoFilter");
        delegatingFilterProxyRegistrationBean.setEnabled(false);
        return delegatingFilterProxyRegistrationBean;
    }