-
-
Notifications
You must be signed in to change notification settings - Fork 271
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Option to specify host for communication with ResourceReaper container #678
Comments
Can you explain your use case more in detail? It sounds like a real edge case. Why do you split or distinguish the container communication? Why can't you treat the Resource Reaper communication as a Docker API communication? What is the advantage? |
Indeed, it is an edge case. I'm working in an enclaved network (private cloud) where there are a lot of security concerns (for example all traffic between resources must be explicitly allowed by firewall rules, encrypted traffic between enclaves). I have a VM with an exposed Docker API that acts as a "hub" for running test containers. Communication to this "hub" comes from different places (e. g. developer machines, build agents) which don't have Docker on them, hence we're going this "hub" way. The Docker API is exposed with additional authentication and the VM is put behind a load balancer (actually we can have multiple VMs in this "hub") which also gives us SSL offloading – basically communication with the Docker API is secured/restricted and limited on a specific port. Communication with the containers on the other hand is way less strict (since they're short lived and only used for integration testing) – we communicate with the VM directly. This is why we differentiate between API and container communication. Hopefully this explains my situation a bit. |
I forgot to mention that we've had this setup running successfully for quite some time now, although we had Resource Reaper disabled and were using a pretty basic cron job to clean up the containers up until now. We'd like to take advantage of Resource Reaper. Thinking about this further, it would even make more sense if I could start the testcontainers/moby-ryuk container manually, keep it always running and then on the code side have something like |
You can start your own instances of Ryuk, but Ryuk requires a network connection to stay alive.
Keeping Ryuk running is probably not difficult, it just requires a durable connection, but it won't remove resources then. Ryuk removes the labeled resources when the connection drops (test session completes). Furthermore, you need to make sure that Ryuk runs on the same Docker host as your resources (not sure if that works with the load balancer in between). The configuration is pretty complex. I am still thinking of how a support of such a complex and edge case might look like. For now, I really think you will have much less pain if you allow TCP traffic to Ryuk. |
Sorry, I still don't understand your use case here @BenasB. You said:
The Ryuk container is also short-lived, it is bound to the lifecycle of the test session (by design). Every interaction with it should look similar to any other container interaction. So why do you have to distinguish here? |
Hi @kiview, yes, that's exactly right - the Ryuk container is also short-lived and I would like it to be treated as other test containers in my system. The distinguishing is not between Ryuk and other test containers, but between the Docker API (making the request to start/stop/orchestrate the containers) and the test containers themselves (Ryuk included). Right now, the starting of the Ryuk container (request to Docker API) and maintaining connection (request to Ryuk container) use the same hostname/endpoint when making requests Edit: as for the design of Ryuk itself, I now understand that it is not intended to be reused between test processes (what I've though (in my 2nd comment) maybe could've been achieved) |
@kiview and I had a quick chat, probably you are looking for something like TESTCONTAINERS_HOST_OVERRIDE. The environment variable allows to override the |
This is a good point, that is not guaranteed right now.
All in all, I think our edge case setup is a bit too complex and we'll have to stick without having ryuk for now. Thanks for your time (and effort for this library in general) @HofmeisterAn and @kiview. I will be closing this issue now. |
Indeed. That is something your load balancer needs to take care of, I guess. I will add the custom configuration |
I can test it, I can see there's |
The snapshot just contains the custom configuration. Not the necessary Resource Reaper adjustments. I will do them next week and get back to you. |
…VERRIDE and TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE
…VERRIDE and TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE
…VERRIDE and TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE
…VERRIDE and TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE (#695)
Is your feature request related to a problem? Please describe.
Right now (by default), when you start any test container, a resource reaper container starts automatically beforehand. The starting of this container takes the same
IDockerEndpointAuthenticationConfiguration
as the container you're trying to start which specifies the endpoint/hostname, which is fine in my scenario. Then, when actually communicating/sending data over TCP to the resource reaper container, it takes the same host name as in the container starting step, which is not fine for my scenario as that endpoint is used only to create containers/communicate with the Docker API (I communicate with the actual containers in my tests through a different host name).Describe the solution you'd like
It would be great if a user could specify the host name on which to communicate with the resource reaper container. I am just not sure how best to propose it.
ResourceReaperPublicHostPort
already, so there's possibility for aResourceReaperHost
, although I'm not the biggest fan of this static setup..With
builder config (e. g.WithResourceReaperCommunicationHost
) (but that goes against the fact that the resource reaper is shared between the containers)Describe alternatives you've considered
I initially thought that maybe specifying
.WithHostname
when creating my container would help, but theTestcontainersContainer.Hostname
(which is used to determine the communication with resource reaper host) has nothing to do with that and furthermore that would only affect the actual test container but not the resource reaper container since it's created here and the configuration can't be reached from outsideThe text was updated successfully, but these errors were encountered: