Is it worth using testcontainers to set up our unit, integration or e2e tests? #1500
Comments
|
Hello, I am a contributor to the testcontainers-go port. I have also used testcontainers-go for integration tests at work. I would say it is 100% worth it in my use cases. I work on a team where one of our products that we own is a conversational AI tool, with models trained specifically for different clients' use cases. These models are used by a containerized service that needs to communicate with the not only the model server, but also many other dependencies (containerized services) that vary between clients. We have set up our entire integration testing suite for all clients' models with testcontainers-go. This allows us to orchestrate the setup, testing, and tear-down of the entire system (all of the services) in one small code base. And now we feel much safer and confident with merging code thanks to the integration testing suite that testcontainers-go allowed us to build. In addition to the above, since this is Go, we can compile our code down to an executable that will run on the target CI/CD server, so we need no other dependencies besides Docker to run the tests. I hope this helps! |
|
To add an alternative, I had a good experience with https://github.com/ory/dockertest which seems to solve the same problem. |
|
Thanks @ClaytonNorthey92 that helps. @fedepaol do you have any code you can link to that uses |
|
I used it in my previous job, but IIRC the examples are pretty clear: https://github.com/ory/dockertest/tree/v3/examples It uses the docker go lib to spin up what's needed, and an exponential backoff to wait for it to be ready before launching tests |
|
Good to know @fedepaol. Both libraries sound like they could work well for us. @fedepaol or @ClaytonNorthey92 would either of you be up for opening a PR with a small prototype? |
|
@arschles yeah I would like to! I should have time this weekend. For the prototype, what functionality would you absolutely want to see? Reading through the docs, the main thing I think it should cover is:
Is there anything else you would like to see covered in the prototype? |
For this, please note that that script is being replaced by a full fledged go test suite in a PR [1] which should merged soon. [1] #1514 |
@fedepaol ah thank you for the link, I will follow what occurs in that test suite rather than |
Prototype for using testcontainers for dependencies: mongodb, minio, and redis Updated the appvoyer.yml file to run these in the test, rather than skip those tests because the environment variables weren't set
Prototype for using testcontainers for dependencies: mongodb, minio, and redis Updated the appvoyer.yml file to run these in the test, rather than skip those tests because the environment variables weren't set
Prototype for using testcontainers for dependencies: mongodb, minio, and redis Updated the appvoyer.yml file to run these in the test, rather than skip those tests because the environment variables weren't set
|
hey @fedepaol and @arschles , I have a small prototype draft pull request here: I few things to note:
|
Prototype for using testcontainers for dependencies: mongodb, minio, and redis Updated the appvoyer.yml file to run these in the test, rather than skip those tests because the environment variables weren't set
Prototype for using testcontainers for dependencies: mongodb, minio, and redis Updated the appvoyer.yml file to run these in the test, rather than skip those tests because the environment variables weren't set
Prototype for using testcontainers for dependencies: mongodb, minio, and redis Updated the appvoyer.yml file to run these in the test, rather than skip those tests because the environment variables weren't set
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
|
@ClaytonNorthey92 sorry I missed this issue. I made some comments on #1544 about drone io and possibly using GitHub actions for those tests. I think that is a good idea, because we're already thinking about doing some more in #1460. I saw you responded there, so I'll move discussion there. Just wanted to make sure you know I'm not ignoring you 😄 |
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
|
Just my 2 cents: tests have worked quite well for a couple of years now testing "service dependencies" such as Mongo, Redis, and Etcd. And most importantly, our codebase is getting very stable which is gives confidence for a lot of people to use it. So I think we should address the following questions if we want to make this change:
Another important part, is testcontainers-go, from what I can tell, would require access to the docker-sock which might be a security concern. I haven't dug too deep yet |
|
@marwan-at-work personally I think there is no problem with our current setup. We have it pretty well documented and I don't see many people having problems running tests locally. But using testcontainers/similar (like #1544) takes a moving piece out of all of our local test setups. You no longer need docker compose. That makes the local dev experience nicer, and means we can take out documentation. I can't speak for how active the library is being developed. "Vanity metrics" on GH are decent - 457 stars right now, last commit from 8 days ago. Also, it came out of the testcontainers organization that has support for many different languages. TL;DR the idea of testcontainers is popular but the go library seems less active. FWIW https://github.com/ory/dockertest looks to be more active. |
I'd say it takes out the docker-compose but enforces docker...what if I want to run the tests against a local Mongo server but not a dockerized one? :) |
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Yup, you're right. There would be no way to do that. Do you think that's a dealbreaker? I think this is a nice idea because it's less setup work in the common case. That said, in #1544, @ClaytonNorthey92 is adding the ability to turn off the testcontainers-based tests. In pseudocode: if testContainers {
startMongoInDocker()
runMongoStorageTests()
}We can take this one step further and give the option to turn off just the testcontainers code: if testContainers {
startMongoInDocker()
}
runMongoStorageTests()This would also allow us to run our entire test suite in Appveyor, which we were not going to be able to do in #1544 as-is @marwan-at-work @ClaytonNorthey92 what do you think? |
I don't think it's a deal breaker but it's certainly weird to require all of our storage backends to be dockerized or the tests wouldn't run. As for supporting both dockerized and not dockerized environments, I worry about maintenance and ease of onboarding for new contributors. That said, we certainly can try stuff out in an experimental branch and not commit to it. If the complexity is justified by the results, then I'm all in. As for Appveyor, I think if the "storage" tests run in one environment (Drone) they probably don't need to run in another? Adding to the fact that if we end up switching to Github Actions, we might not need Appveyor at all, so we can definitely revisit after our pair programming session ✌️ All of the above are more opinions than anything, so definitely happy to go with whatever the majority thinks. |
@marwan-at-work let's see how this looks in #1544 and see how complex it gets. Are you ok with that? |
|
I have some ideas on how we could make this work. First question though, do we want to default testcontainers to be used during tests? Or do we want to to default to using a user-defined location (ex. locally, remotely)? |
|
@ClaytonNorthey92 let's default to user-defined location (as it is now), and I'll set up the env var in our CI to turn on TestContainers. We should also add docs for why/how to turn testcontainers on How does that sound? |
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis
Prototype for using testcontainers for dependencies: mongodb, minio, and redis squash me: use testcontainers in appveyor use newer vs attempt docker upgrade
Prototype for using testcontainers for dependencies: mongodb, minio, and redis squash me: use testcontainers in appveyor use newer vs attempt docker upgrade
|
Hi folks! Testcontainers Go maintainer here 👋 Is this issue still under consideration? I saw some concerns about maintenance and vanity metrics, I'd say testcontainers-go is very active since 2022, and regarding stars, it seems it's close to dockertest in popularity: https://star-history.com/#testcontainers/testcontainers-go&ory/dockertest&Date If you are interested in knowing about more, please let me know if I can be of help here. 🙏 Cheers! |
Testcontainers is a convenience library to set up containers from Go code. We could use it to set up pre-requisite services like Redis, Mongo, Minio, etc...
The text was updated successfully, but these errors were encountered: