TestContainers Hang After First Run

[ashwin-jacob]

I added a second TestContainer into our class. On the first run, everything will pass with no problems. The second run will hang on the command to permit Docker to bind. In order for it to work, I have to stop the java process, restart docker, and rerun it. At that point, it will work on the first run but not the second run again.

Here are the logs where I got the issue from:

09:01:42.591 [main] DEBUG 🐳 [localstack/localstack:latest] - Starting container: localstack/localstack:latest
09:01:42.592 [main] DEBUG 🐳 [localstack/localstack:latest] - Trying to start container: localstack/localstack:latest (attempt 1/1)
09:01:42.592 [main] DEBUG 🐳 [localstack/localstack:latest] - Starting container: localstack/localstack:latest
09:01:42.592 [main] INFO 🐳 [localstack/localstack:latest] - Creating container for image: localstack/localstack:latest
09:01:42.592 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: localstack/localstack:latest at registry: index.docker.io
09:01:42.592 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Cached auth found: [AuthConfig{username=XXXX, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]
09:01:42.592 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=XXXX, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]
09:01:42.596 [main] DEBUG org.testcontainers.utility.PrefixingImageNameSubstitutor - No prefix is configured
09:01:42.596 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for alpine:3.5 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
09:01:42.596 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: alpine:3.5
09:01:42.602 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: alpine:3.5 at registry: index.docker.io
09:01:42.602 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Cached auth found: [AuthConfig{username=XXXX, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]
09:01:42.602 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=XXXX, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]
09:01:42.611 [main] DEBUG org.testcontainers.utility.MountableFile - Copying classpath resource(s) from jar:file:/Users/XXXXX/.m2/repository/org/testcontainers/testcontainers/1.15.3/testcontainers-1.15.3.jar!/org/testcontainers/utility/ResourceReaper.class to /tmp/.testcontainers-tmp-3628711180189158291 to permit Docker to bind
09:01:42.611 [main] DEBUG org.testcontainers.utility.MountableFile - Copying resource org/testcontainers/utility/ResourceReaper.class from JAR file /Users/XXXXX/.m2/repository/org/testcontainers/testcontainers/1.15.3/testcontainers-1.15.3.jar
09:01:42.626 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl@27d57a2c[name=<null>,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,healthcheck=<null>,argsEscaped=<null>,entrypoint=<null>,image=alpine:3.5,volumes=com.github.dockerjava.api.model.Volumes@2ff95fc6,workingDir=<null>,macAddress=<null>,onBuild=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@9d1a267,stopSignal=<null>,stopTimeout=<null>,hostConfig=HostConfig(binds=[/private/tmp/.testcontainers-tmp-3628711180189158291:/dummy:ro], blkioWeight=null, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, memorySwappiness=null, nanoCPUs=null, capAdd=null, capDrop=null, containerIDFile=null, cpuPeriod=null, cpuRealtimePeriod=null, cpuRealtimeRuntime=null, cpuShares=null, cpuQuota=null, cpusetCpus=null, cpusetMems=null, devices=null, deviceCgroupRules=null, deviceRequests=null, diskQuota=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=null, groupAdd=null, ipcMode=null, cgroup=null, links=[], logConfig=LogConfig(type=null, config=null), lxcConf=null, memory=null, memorySwap=null, memoryReservation=null, kernelMemory=null, networkMode=null, oomKillDisable=null, init=null, autoRemove=null, oomScoreAdj=null, portBindings=null, privileged=null, publishAllPorts=null, readonlyRootfs=null, restartPolicy=null, ulimits=null, cpuCount=null, cpuPercent=null, ioMaximumIOps=null, ioMaximumBandwidth=null, volumesFrom=null, mounts=null, pidMode=null, isolation=null, securityOpts=null, storageOpt=null, cgroupParent=null, volumeDriver=null, shmSize=null, pidsLimit=null, runtime=null, tmpFs=null, utSMode=null, usernsMode=null, sysctls=null, consoleSize=null),labels={org.testcontainers=true, org.testcontainers.sessionId=d5b6c43c-6fcc-4ea6-aa82-0b897b40917e},shell=<null>,networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=AuthConfig(username=XXXX, email=null, registryAddress=index.docker.io, stackOrchestrator=null)]
09:01:42.701 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 2e6a5329860ca4325e1ea4c4238a43569a83772ad36caeedd2b8bd2bdb7f886e

• Docker Desktop: 3.5.1
• Docker Engine: 20.10.7
• TestContainers Version: 1.15.3

Here is a snippet of our class

@SpringBootTest
@Testcontainers
class RandomTest {
    private static final DockerImageName localstackImage = DockerImageName.parse("localstack/localstack:latest");

    @Container
    private static final LocalStackContainer localStackContainer = new LocalStackContainer(localstackImage)
            .withServices(LocalStackContainer.Service.SQS);
    
    @Container
    private static final MongoDBContainer mongoDBContainer = new MongoDBContainer(DockerImageName.parse("mongo:4.4.6-xenial"));

    @DynamicPropertySource
    static void properties(DynamicPropertyRegistry dynamicPropertyRegistry) {
        dynamicPropertyRegistry.add("SQS_SERVICE_ENDPOINT", () -> localStackContainer.getEndpointConfiguration(LocalStackContainer.Service.SQS).getServiceEndpoint());
    }
}

[ashwin-jacob]

Not sure if this helps but I am running this with 1.15.3 of LocalStackContainer and MongoDBContainer

[ShawnRG]

I currently have the same issue on my MacBook Pro (Big Sur 11.6).
For some reason the container running the alpine images stays in its CREATED state. Attempting to delete the container does not work, I have to - like @ashwin-jacob said - restart my docker engine for it to work again.

I am running on Testcontainers 1.16.0

[dbyron-sf]

Happening to me as well. Similar setup to @ShawnRG. I'm running Docker Desktop 4.1.0 (69386) and Docker Engine 20.10.8.

[kiview]

Hi @ashwin-jacob, which OS are you using?

Can we try to triage if the issue is related to the LocalStackContainer?
Please verify if you can reproduce the same issue with this GenericContainer based example:
https://github.com/testcontainers/testcontainers-java-repro/blob/master/src/test/java/org/testcontainers/repro/ReproExampleTest.java

If this does not occur with GenericContainer, please provide a reproducer based on the setup and the used containers, ideally as a plain Java example without any framework code.

Everyone else reporting on this issue, please report your OS and your if you use a special architecture (e.g. M1 Macbook).

[ShawnRG]

The example works fine for me, even in combination with Spring Boot's integration test.

An important side-note here is that it failed for me when using a BrowserWebDriverContainer.

[bsideup]

For some reason the container running the alpine images stays in its CREATED state

This sounds like a bug in Docker, not Testcontainers, as we do not control the transitioning from CREATED to RUNNING.
If CreateContainerCmd gets stuck, this would mean that there is something in Docker's API that blocks it from returning container's information, and we cannot proceed without it.

Attempting to delete the container does not work

This is also a strong signal that something is terribly wrong inside the Docker daemon (deadlock? File system-related issues?)

Have you reported it to Docker yet?

[ShawnRG]

Thanks for the info, I will report an issue with docker as well!

[kiview]

Are you using BrowserWebDriverContainer with Chrome or Firefox? There was a very specific issue with the new Chrome image on certain systems, see #4315.

[ShawnRG]

I am using it with Chrome.
Just did a test run with Firefox, but the issue still persists.

[ShawnRG]

Update:
I have completely purged and re-installed Docker from my system using the uninstall feature in Docker Desktop and removing it via Homebrew.

Now I do not get the issue anoymore, I have tested two previously failing test scenarios and they seem to be working consistently now.

[dbyron-sf]

This issue disappeared for me after a reboot.

[kiview]

Since this seemed to be some Docker-related issues, I will close this for now. If you have any new information, feel free to update this issue accordingly.