[Bug]: 1.18.1 to 1.18.3 gives Out Of Heap Memory exception for DockerComposeContainer #7239

frankjkelly · 2023-06-22T21:57:09Z

Module

Core

Testcontainers version

1.18.1 to 1.18.3

Using the latest Testcontainers version?

Yes

Host OS

MacOS

Host Arch

x86 Intel Core i7

Docker version

$ docker version
Client:
 Cloud integration: v1.0.31
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:12:52 2023
 OS/Arch:           darwin/amd64
 Context:           default

Server: Docker Desktop 4.19.0 (106363)
 Engine:
  Version:          23.0.5
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       94d3ad6
  Built:            Wed Apr 26 16:17:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

When upgrading only test containers from 1.18.0 to 1.18.1 got heap error on startup of my Docker Compose based Integration tests.
Also tried 1.18.3.

Relevant log output

org.gradle.api.internal.tasks.testing.TestSuiteExecutionException: Could not complete execution for Gradle Test Executor 1.
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:64)
	at java.base@11.0.8/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@11.0.8/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base@11.0.8/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@11.0.8/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy5.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
	at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
	at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
	at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
	at org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:48)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:244)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:92)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:86)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:122)
	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:462)
	at java.base/java.io.InputStream.transferTo(InputStream.java:705)
	at java.base/java.nio.file.Files.copy(Files.java:3119)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:362)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.transferTo(MountableFile.java:327)
	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:306)
	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:282)
	at org.testcontainers.containers.GenericContainer$$Lambda$482/0x0000000800420840.accept(Unknown Source)
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:430)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
	at org.testcontainers.containers.GenericContainer$$Lambda$447/0x0000000800393c40.call(Unknown Source)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at org.testcontainers.containers.ContainerisedDockerCompose.invoke(DockerComposeContainer.java:727)
	at org.testcontainers.containers.DockerComposeContainer.runWithCompose(DockerComposeContainer.java:339)
	at org.testcontainers.containers.DockerComposeContainer.createServices(DockerComposeContainer.java:260)

Additional Information

No response

Saimurugeshwari · 2023-06-28T20:57:37Z

i would like to contribute this issue, kindly assign this to me

linus-learns · 2023-07-06T09:50:15Z

This was caused by 00a509c

eddumelendez · 2023-07-06T10:15:21Z

is your docker-compose file that big?

linus-learns · 2023-07-06T10:59:51Z

If I interpret the code correctly, you are trying to copy the whole directory where the compose file is located into the container causing the oom:
final String pwd = dockerComposeBaseFile.getAbsoluteFile().getParentFile().getAbsolutePath();
final String containerPwd = convertToUnixFilesystemPath(pwd);
...
withCopyFileToContainer(MountableFile.forHostPath(pwd), containerPwd);

from

testcontainers-java/core/src/main/java/org/testcontainers/containers/ContainerisedDockerCompose.java

Line 33 in 595076c

    
           final String pwd = dockerComposeBaseFile.getAbsoluteFile().getParentFile().getAbsolutePath();

and onwards

eddumelendez · 2023-07-11T16:57:01Z

Thanks! good catch! I didn't fall into that because the docker-compose files are inside different folders.

I need to figure out how to fix it properly. The workaround would be to move the docker-compose.yml to a folder

markush81 · 2023-08-22T14:27:53Z

Depending on what your docker-compose.yml contains, it might be necessary to copy stuff. As an example if you mount config files for a container. So a mechanism like .dockerignore could be of help.

As workaround use withLocalCompose(true) if possible.

jandry · 2023-09-06T09:29:23Z

Still have issue on 1.19.0. Only working with 1.18.0

frankjkelly · 2023-09-06T12:51:14Z

So sounds like two work-arounds so far

Move docker-compose file to dedicated folder as per [Bug]: 1.18.1 to 1.18.3 gives Out Of Heap Memory exception for DockerComposeContainer #7239 (comment)
Use use withLocalCompose(true) as per [Bug]: 1.18.1 to 1.18.3 gives Out Of Heap Memory exception for DockerComposeContainer #7239 (comment)

Michael-Rosa-Imprivata · 2023-09-12T23:09:01Z

@frankjkelly - fancy meeting you here, long time no see :).
So i ran into the same problem, but we are on version 1.19.0 and withLocalCompose(true) did not do the trick for me, but we have a developer investigating both work arounds further. Curious if you had any luck on your end with the workarounds?

frankjkelly · 2023-09-13T12:40:39Z

Hey there @Michael-Rosa-Imprivata - wow fancy meeting you here!? Hope all is well.
I haven't tried any of the workarounds yet so we just pinned the testcontainers version :-(

frankjkelly · 2023-09-13T14:36:20Z

FYI I just tried work-around of moving the docker-compose.yml to it's own directory but still no luck

Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image docker/compose:1.29.2
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:361)
	at app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:334)
	at app//org.testcontainers.containers.ContainerisedDockerCompose.invoke(ContainerisedDockerCompose.java:64)
	at app//org.testcontainers.containers.ComposeDelegate.runWithCompose(ComposeDelegate.java:254)
	at app//org.testcontainers.containers.ComposeDelegate.createServices(ComposeDelegate.java:163)
	at app//org.testcontainers.containers.DockerComposeContainer.start(DockerComposeContainer.java:137)
	at app//com.cogito.platform.signal.AbstractIntegrationTest.<clinit>(AbstractIntegrationTest.java:102)
	... 97 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:346)
	... 103 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:565)
	at app//org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:356)
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 104 more
Caused by: java.lang.IllegalStateException: Container did not start correctly.
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:497)
	... 106 more

Michael-Rosa-Imprivata · 2023-09-28T14:54:43Z

Yea same here Frank. Work arounds are not doing the trick. We decided to revert back to the previous working version as well for now.

markush81 · 2023-09-29T15:56:29Z

@Michael-Rosa-Imprivata just out of interest

So withLocalCompose(true) still runs into java.lang.OutOfMemoryError: Java heap space.

That sounds odd (or different issue), since the LocalDockerCompose doesn't copy, compared to ContainerisedDockerCompose in GenericContainer.

Could you paste stacktrace/output/code snippet, when using withLocalCompose(true).

jandry · 2023-10-04T10:36:13Z

Hi,
I tried a workaround : I forced a downgrade org.apache.commons:commons-compress to 1.22.0. But it's seems not working also.
Still only working workaround, stay with 1.18.0 :(

Exemple in gradle

fun Configuration.resolutionStrategyForSecurity() {
    resolutionStrategy.eachDependency {
        if (requested.group == "org.apache.commons"  && (requested.name == "commons-compress") && (requested.version == "1.23.0" || requested.version == "1.24.0")) {
            useVersion("1.22")
            because("Memory issue in integration test or system test with testcontainers 1.18.3/1.19.0 (java.lang.OutOfMemoryError: Java heap space in ContainerState.copyFileToContainer) https://github.com/testcontainers/testcontainers-java/issues/2863")
        }
    }
}

eugene-kuntsevich · 2023-10-05T12:20:21Z

withLocalCompose(true) helped to migrate to 1.19.1. Could somebody, please, explain what property withLocalCompose doing?

markush81 · 2023-10-05T13:01:42Z

@eugene-kuntsevich https://java.testcontainers.org/modules/docker_compose/#local-compose-mode

In short, it executes docker-compose (or v2 docker compose) on your machine (or the CI machine) instead of doing it, in a container - and therefore has no need to copy stuff.

frankjkelly · 2023-10-05T13:23:43Z

When I try withLocalCompose(true)( with 1.18.0 and 1.19.1) I get

Caused by: java.lang.ExceptionInInitializerError
	at com.cogito.platform.admin.AbstractIntegrationTest$ContextConfiguration.<clinit>(AbstractIntegrationTest.java:136)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at org.springframework.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:602)
	at org.springframework.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:363)
	at org.springframework.cglib.proxy.Enhancer.generate(Enhancer.java:585)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:110)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:108)
	at org.springframework.cglib.core.internal.LoadingCache$2.call(LoadingCache.java:54)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:61)
	... 86 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image alpine/socat:1.7.4.3-r0
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:349)
	at app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at app//org.testcontainers.containers.DockerComposeContainer.startAmbassadorContainers(DockerComposeContainer.java:359)
	at app//org.testcontainers.containers.DockerComposeContainer.start(DockerComposeContainer.java:190)
	at app//com.cogito.platform.admin.AbstractIntegrationTest.<clinit>(AbstractIntegrationTest.java:101)
	... 97 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
	... 101 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:553)
	at app//org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 102 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Aborting attempt to link to container tcntmtj1rvmc_entity-service_1 as it is not running
	at app//org.testcontainers.containers.GenericContainer.applyConfiguration(GenericContainer.java:839)
	at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:378)
	... 104 more

Is there a second part to this workaround that is needed?

jandry · 2023-10-05T18:36:01Z

withLocalCompose(true) helped to migrate to 1.19.1. Could somebody, please, explain what property withLocalCompose doing?

Like explained here https://java.testcontainers.org/modules/docker_compose/, it means that testcontainers use the local binary of docker compose instead of using a docker image with docker compose inside.
So locally no issue, but in CI this means that your runner image must have a docker compose binary

Gregory-Berkman-Imprivata · 2023-10-20T19:23:38Z

Any update on this?

odwrotnie · 2023-11-17T21:10:31Z

Any update?

erwin-ramirez-da · 2024-01-11T10:44:12Z

Is there any workaround for this?

alexanderankin · 2024-01-26T04:32:58Z

#1348 is i think the most highly relevant piece - if copying the files is really non-negotiable (maybe making this bind vs copy behavior configurable is agreeable to testcontainers-java maintainers?) then implementing .dockerignore support in docker-java is probably the way to go.

main...alexanderankin:testcontainers-java:reduce_7239

gba-foundever · 2024-01-27T11:27:44Z

Hi @alexanderankin , thanks a lot !
I tried your fix build a snapshot version locally and it working fine for me

findepi · 2024-01-30T19:35:42Z

#2864 was replacing some in-memory buffering (ByteArrayOutputStream) with a temp file. Not sure if it still is relevant.

eddumelendez · 2024-07-04T22:05:50Z

#8409 has been merged and it should fix the issue.

frankjkelly added the type/bug label Jun 22, 2023

eddumelendez added the area/docker-compose label Oct 2, 2023

This was referenced Oct 4, 2023

GenericContainer#withCopyFileToContainer is not efficient memory-wise #4203

Closed

Huge MountableFiles result in OutOfMemoryError #1348

Closed

OutOfMemoryError when copyFileToContainer used on large file #2863

Closed

alexanderankin added a commit to alexanderankin/testcontainers-java that referenced this issue Jan 26, 2024

testcontainers#7239 - reduce the pain by making it configurable

21cbc05

chenlin2001 mentioned this issue Feb 15, 2024

Podman with Docker-compose not working #5351

Open

mgoworko mentioned this issue May 17, 2024

[RORDEV-1152] ES testcontainer fixes sscarduzio/elasticsearch-readonlyrest-plugin#1018

Merged

eddumelendez closed this as not planned Won't fix, can't repro, duplicate, stale Jul 4, 2024

frankjkelly mentioned this issue Jul 5, 2024

Fix copy large files #8409

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: 1.18.1 to 1.18.3 gives Out Of Heap Memory exception for DockerComposeContainer #7239

[Bug]: 1.18.1 to 1.18.3 gives Out Of Heap Memory exception for DockerComposeContainer #7239

frankjkelly commented Jun 22, 2023

Saimurugeshwari commented Jun 28, 2023

linus-learns commented Jul 6, 2023

eddumelendez commented Jul 6, 2023

linus-learns commented Jul 6, 2023 •

edited

Loading

eddumelendez commented Jul 11, 2023

markush81 commented Aug 22, 2023 •

edited

Loading

jandry commented Sep 6, 2023

frankjkelly commented Sep 6, 2023

Michael-Rosa-Imprivata commented Sep 12, 2023 •

edited

Loading

frankjkelly commented Sep 13, 2023

frankjkelly commented Sep 13, 2023

Michael-Rosa-Imprivata commented Sep 28, 2023

markush81 commented Sep 29, 2023 •

edited

Loading

jandry commented Oct 4, 2023 •

edited

Loading

eugene-kuntsevich commented Oct 5, 2023

markush81 commented Oct 5, 2023 •

edited

Loading

frankjkelly commented Oct 5, 2023

jandry commented Oct 5, 2023

Gregory-Berkman-Imprivata commented Oct 20, 2023

odwrotnie commented Nov 17, 2023

erwin-ramirez-da commented Jan 11, 2024

alexanderankin commented Jan 26, 2024 •

edited

Loading

gba-foundever commented Jan 27, 2024

findepi commented Jan 30, 2024

eddumelendez commented Jul 4, 2024

[Bug]: 1.18.1 to 1.18.3 gives Out Of Heap Memory exception for DockerComposeContainer #7239

[Bug]: 1.18.1 to 1.18.3 gives Out Of Heap Memory exception for DockerComposeContainer #7239

Comments

frankjkelly commented Jun 22, 2023

Module

Testcontainers version

Using the latest Testcontainers version?

Host OS

Host Arch

Docker version

What happened?

Relevant log output

Additional Information

Saimurugeshwari commented Jun 28, 2023

linus-learns commented Jul 6, 2023

eddumelendez commented Jul 6, 2023

linus-learns commented Jul 6, 2023 • edited Loading

eddumelendez commented Jul 11, 2023

markush81 commented Aug 22, 2023 • edited Loading

jandry commented Sep 6, 2023

frankjkelly commented Sep 6, 2023

Michael-Rosa-Imprivata commented Sep 12, 2023 • edited Loading

frankjkelly commented Sep 13, 2023

frankjkelly commented Sep 13, 2023

Michael-Rosa-Imprivata commented Sep 28, 2023

markush81 commented Sep 29, 2023 • edited Loading

jandry commented Oct 4, 2023 • edited Loading

eugene-kuntsevich commented Oct 5, 2023

markush81 commented Oct 5, 2023 • edited Loading

frankjkelly commented Oct 5, 2023

jandry commented Oct 5, 2023

Gregory-Berkman-Imprivata commented Oct 20, 2023

odwrotnie commented Nov 17, 2023

erwin-ramirez-da commented Jan 11, 2024

alexanderankin commented Jan 26, 2024 • edited Loading

gba-foundever commented Jan 27, 2024

findepi commented Jan 30, 2024

eddumelendez commented Jul 4, 2024

linus-learns commented Jul 6, 2023 •

edited

Loading

markush81 commented Aug 22, 2023 •

edited

Loading

Michael-Rosa-Imprivata commented Sep 12, 2023 •

edited

Loading

markush81 commented Sep 29, 2023 •

edited

Loading

jandry commented Oct 4, 2023 •

edited

Loading

markush81 commented Oct 5, 2023 •

edited

Loading

alexanderankin commented Jan 26, 2024 •

edited

Loading