Epic: small footprint container usable for development, testing and production purposes #5292

poikilotherm · 2018-11-08T12:53:16Z

IMHO this is an epic, not a single story.

This issue is successor to #5187 and closes it.

It is intended to serve as a base for solutions or make life easier in:

Running integration tests for more than just API #5068 - Running integration tests for more than just API
installing Dataverse under Payara 5 #4172 - Payara 5 Upgrade
Refactoring of S3AccessIO driver #5072 - Refactoring of S3 storage driver
Docker for production #4665 - Docker in production and projects related to that (IQSS/dataverse-docker and the like)
Get Dataverse running on OpenShift (Docker and Kubernetes) #4040 - OpenShift / Kubernetes deployments

This is blocked by some stuff and relies on some prior art to be done:

~~Dependency Housekeeping and how developers will benefit from it #5288 - Dependency Housekeeping~~ Done!
Stripping AWS Mail component with TrueZIP blocks packaging for docker images #5274 - Stripping AWS dependency and TrueZIP from POM
Dependency Housekeeping, Round 2 #5360 - Dependency Housekeeping, second round
As an admin and developer, I'd benefit from a simplified configuration mechanism #5293 - Simplify configuration of the application.
Proposal: make persistent EJB timers non-persistent or not rely on database #5345 - Make EJB timers non-persistent to drop database requirement.
Bootstrapping config on first deployment #5361 - Bootstraping empty instance
Eating 100% CPU when setting com.sun.enterprise.server.logging.GFFileHandler.logtoFile=false payara/Payara#3506 and payara/docker-payaraserver-full#72 - Logging to console eats 1 CPU

Things to consider:

Add a HEALTCHECK CMD that uses a to-be-built API endpoint reporting on health status?

Vision / Proposal

Currently, when running integration tests or deploying Dataverse to Docker/Kubernetes, only kind of heavyweight solutions exist with the DockerAIO for IT tests and most (all?) Docker/Kubernetes/OpenShift approaches relying on the installer script.

I encourage the following vision:

Build a new image directly from a Maven target, needing only a dev env plus Docker installed and running (obsolete once img support is in place...)
Make this image as small as possible with an application server only, add dependencies and the application.
Anything else lives in other containers, following the micro services credo.
Make the application container itself stateless, also following the micro services credo. (This does not affect the use of volumes/...)
Make the configuration a breeze - don't use the install script inside the container. Instead provide options to get the configuration inside from external sources.

To get there I suggest using:

fabric8io/docker-maven-plugin
payara/server-full Docker image
Separate Maven profile to build this as long as the "old way" is still used broadly.

Things to keep in mind:

Ideally this is based on Payara 5, not 4.
Let users still use the "old" WAR file approach in parallel! Somebody might rely on that. (That's why I killed As a developer and sysadmin, I would enjoy the benefits of using Java MicroProfile based deployments with an application server like Payara Micro or Thorntail. #5187)
Let the configuration ways currently know to all users still work. Somebody might rely on that!

Give it a shot! (Testing)

To test, just have Docker, Maven, Git and Java installed.
Then do:

git clone https://github.com/poikilotherm/dataverse -b 5292-small-container
mvn -Pcontainer clean package docker:build docker:run -DskipTests

Please keep in mind that this is a feature branch. If you already have a cloned dataverse repo, you might better off using:

git remote add poikilotherm https://github.com/poikilotherm/dataverse
git fetch poikilotherm 5292-small-container
git checkout -b 5292-small-container

I regularly update this feature branch to be based on the latest develop. This involves rebasing, which will let your local branch be diverged. In that case, simply use git reset --hard poikilotherm/5292-small-container after a fetch.

The text was updated successfully, but these errors were encountered:

This commit adds initial support for building an "application only" container. That is achived by adding the fabric8io/docker-maven-plugin to a new Maven profile called "container". Build via `mvn docker:build`. The build is based on the upstream Payara 5 image right now. This is a TODO, as the upstream container project is not very responsive and there is no proper signal handling in place.

poikilotherm · 2018-11-09T13:41:31Z

@pdurbin and other: initial work on the building part has been to my feature branch.

Right now this will (of course) not work. The Postgres driver is missing right now and the config part (see #5293) has to be addressed first.

~~As I wrote in the commit message, the upstream container project is in some parts not very responsive/active (see here, here, here and here).~~

~~Will try my best to get things upstream, but maybe better fork and try to get this upstream later:~~

Fix the missing signal handling including gracefull shutdowns
~~Get inspiration from the stuff that @shoeper did~~
~~Adress more issues from upstream.~~

Also wondering if automated builds and security scans from quay.io could be interesting for this.

poikilotherm · 2018-11-13T14:37:15Z

I am currently working on an updated payara image, see https://github.com/poikilotherm/docker-payaraserver-full/tree/refactor . Will try to get my work upstream, just a few hours ago they merged stuff :-D

Since the upstream merged a lot of stuff cleaning up most of the issues (see here), I switched back to use those. Some issues still exist with the init system.

This commit adds initial support for building an "application only" container. That is achived by adding the fabric8io/docker-maven-plugin to a new Maven profile called "container". Build via `mvn docker:build`. The build is based on the upstream Payara 5 image right now. This is a TODO, as the upstream container project is not very responsive and there is no proper signal handling in place.

poikilotherm · 2018-11-22T12:58:50Z

I just opened PR payara/docker-payaraserver-full#61 and hope things will get merged.

This commit adds initial support for building an "application only" container. That is achived by adding the fabric8io/docker-maven-plugin to a new Maven profile called "container". Build via `mvn docker:build`. The build is based on the upstream Payara 5 image right now. This is a TODO, as the upstream container project is not very responsive and there is no proper signal handling in place.

poikilotherm · 2020-10-23T19:14:15Z

Current niceness blockers:

It would be a great relief not to include the bootstrap shell scripts but make Dataverse do it themself. Bootstrapping config on first deployment #5361, scripts for rebuilding a dev environment (with sample data) #7256
It would so great to not fiddle with the Postgres driver, but include it in the WAR. Include Postgres JDBC driver in pom.xml #6819
It would be so nice not to create a tedious init script configuring Dataverse on container boot. This is true for multiple resources.
A start to be made with the DB connection, using @DataSourceDefinition, see Include Postgres JDBC driver in pom.xml #6819. JMS etc to follow.
No more fiddling with password aliases by using secrets via MicroProfile Config. As a sysadmin I want to use MicroProfile Config API to configure my installation #7000, As an admin and developer, I'd benefit from a simplified configuration mechanism #5293
No more fiddling with configuration parsing during startup by using MicroProfile Config. As a sysadmin I want to use MicroProfile Config API to configure my installation #7000
Some other, smaller things to do...

But maybe we can just deal with it for now. Jenkinsfiles missing. Solr image missing.

Instead of using the upstream Payara image we build on the OpenJDK image which is also used by Solr. That way we reduce image pulls. We also optimize the domain1 configuration to be more production ready. Relates to IQSS#5292

1. Moving some specifics from the startup script to the build time config, saving time during startup. 2. Enlarging thread pools, setting better options for memory management and many other optimizations for using in production. See also gdcc/dataverse-kubernetes#218 Relates to IQSS#5292

…un & start targets, etc. IQSS#5292

…Solr and Postgres in single Maven call. IQSS#5292

Instead of using the upstream Payara image we build on the OpenJDK image which is also used by Solr. That way we reduce image pulls. We also optimize the domain1 configuration to be more production ready. Relates to IQSS#5292

1. Moving some specifics from the startup script to the build time config, saving time during startup. 2. Enlarging thread pools, setting better options for memory management and many other optimizations for using in production. See also gdcc/dataverse-kubernetes#218 Relates to IQSS#5292

…un & start targets, etc. IQSS#5292

…Solr and Postgres in single Maven call. IQSS#5292

…ntainer scripts. IQSS#5292

poikilotherm · 2021-06-24T15:57:51Z

There is still plenty of stuff to do. Yet this issue is quite dated and it's starting to become messy. Closing this for now.

Note, that there is traction at https://github.com/gdcc/dataverse/tree/develop+ct, https://github.com/gdcc/dataverse-kubernetes and much MPCONFIG stuff going on.

Instead of using the upstream Payara image we build on the OpenJDK image which is also used by Solr. That way we reduce image pulls. We also optimize the domain1 configuration to be more production ready. Relates to IQSS#5292

1. Moving some specifics from the startup script to the build time config, saving time during startup. 2. Enlarging thread pools, setting better options for memory management and many other optimizations for using in production. See also gdcc/dataverse-kubernetes#218 Relates to IQSS#5292

…un & start targets, etc. IQSS#5292

…Solr and Postgres in single Maven call. IQSS#5292

…ntainer scripts. IQSS#5292

This was referenced Nov 9, 2018

Dependency Housekeeping and how developers will benefit from it #5288

Closed

Provide dev Dataverse deployment whole-tale/whole-tale#49

Closed

Where to find images on Docker Hub IQSS/dataverse-docker#8

Open

craig-willis mentioned this issue Nov 16, 2018

Docker for production #4665

Closed

poikilotherm mentioned this issue Nov 19, 2018

Do not require setup of PID credentials for development environments #5327

Closed

This was referenced Nov 26, 2018

JMS resources from glassfish-resources.xml not usable - JNDI lookup fails payara/Payara#3449

Closed

Proposal: make persistent EJB timers non-persistent or not rely on database #5345

Closed

poikilotherm mentioned this issue Dec 3, 2018

Bootstrapping config on first deployment #5361

Closed

poikilotherm mentioned this issue Jan 4, 2019

installing Dataverse under Payara 5 #4172

Closed

poikilotherm mentioned this issue Mar 1, 2019

Increasingly slow feedback loop for developers, increasingly large WAR files #5593

Closed

This was referenced May 29, 2019

Upgrade Glassfish from 4.1 #2628

Closed

Create container image using Payara 5 gdcc/dataverse-kubernetes#63

Closed

poikilotherm mentioned this issue Jul 13, 2020

Include Postgres JDBC driver in pom.xml #6819

Closed

poikilotherm mentioned this issue Aug 31, 2020

5292 mvn container #7245

Closed

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Oct 23, 2020

Update docker-maven-plugin to 0.34.1. IQSS#5292

f5d28f3

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Mar 26, 2021

refactor(container): reunite the docker images to enable usage with r…

7cb10c0

…un & start targets, etc. IQSS#5292

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Mar 26, 2021

feature(container): add simple run configuration to start Dataverse, …

68f439f

…Solr and Postgres in single Maven call. IQSS#5292

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Jun 11, 2021

Update docker-maven-plugin to 0.34.1. IQSS#5292

8ed0d63

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Jun 11, 2021

refactor(container): reunite the docker images to enable usage with r…

6d6b9d1

…un & start targets, etc. IQSS#5292

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Jun 11, 2021

feature(container): add simple run configuration to start Dataverse, …

e94024c

…Solr and Postgres in single Maven call. IQSS#5292

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Jun 11, 2021

build(container): update to Docker Maven Plugin 0.36.0 IQSS#5292

e9f691e

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Jun 11, 2021

style(container): add missing license informations and sources for co…

d78e463

…ntainer scripts. IQSS#5292

poikilotherm closed this as completed Jun 24, 2021

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 23, 2021

Update docker-maven-plugin to 0.34.1. IQSS#5292

040ea8e

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 23, 2021

refactor(container): reunite the docker images to enable usage with r…

5979ea0

…un & start targets, etc. IQSS#5292

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 23, 2021

feature(container): add simple run configuration to start Dataverse, …

8c06cd0

…Solr and Postgres in single Maven call. IQSS#5292

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 23, 2021

build(container): update to Docker Maven Plugin 0.36.0 IQSS#5292

e9cb67f

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 23, 2021

style(container): add missing license informations and sources for co…

954e3af

…ntainer scripts. IQSS#5292

poikilotherm mentioned this issue Nov 22, 2021

Increase testability by using Testcontainers #8250

Closed

This was referenced Aug 24, 2022

Feature Request/Idea: create a base container image providing a Dataverse-tuned Payara application server #8932

Closed

Feature Request/Idea: provide a Dataverse container image #8934

Closed

GPortas mentioned this issue Mar 3, 2023

Dataverse container-based development environment #9423

Closed

pdurbin changed the title ~~EPIC: small footprint container usable for development, testing and production purposes~~ Epic: small footprint container usable for development, testing and production purposes Mar 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Epic: small footprint container usable for development, testing and production purposes #5292

Epic: small footprint container usable for development, testing and production purposes #5292

poikilotherm commented Nov 8, 2018 •

edited

Loading

poikilotherm commented Nov 9, 2018 •

edited

Loading

poikilotherm commented Nov 13, 2018 •

edited

Loading

poikilotherm commented Nov 22, 2018 •

edited

Loading

poikilotherm commented Oct 23, 2020 •

edited

Loading

poikilotherm commented Jun 24, 2021 •

edited

Loading

Epic: small footprint container usable for development, testing and production purposes #5292

Epic: small footprint container usable for development, testing and production purposes #5292

Comments

poikilotherm commented Nov 8, 2018 • edited Loading

Vision / Proposal

Give it a shot! (Testing)

poikilotherm commented Nov 9, 2018 • edited Loading

poikilotherm commented Nov 13, 2018 • edited Loading

poikilotherm commented Nov 22, 2018 • edited Loading

poikilotherm commented Oct 23, 2020 • edited Loading

poikilotherm commented Jun 24, 2021 • edited Loading

poikilotherm commented Nov 8, 2018 •

edited

Loading

poikilotherm commented Nov 9, 2018 •

edited

Loading

poikilotherm commented Nov 13, 2018 •

edited

Loading

poikilotherm commented Nov 22, 2018 •

edited

Loading

poikilotherm commented Oct 23, 2020 •

edited

Loading

poikilotherm commented Jun 24, 2021 •

edited

Loading