Provide Docker file to create a dev environment

[scolapasta]

Step 1: install basic services: postgrsql, solr, glassfish, etc.
Step 2: Run equivalent of installer script
Step 3: ???
Step 4: Profit!

[scolapasta]

Step 0: Investigate what community has already done in this direction.

[stevenmce]

It may be worthwhile looking at this for a production environment as well. We are considering this for ADA. Cheers Steve (@stevenmce)

…

Sent from my iPhone On 21 Jun 2017, at 5:53 am, Gustavo Durand <notifications@github.com<mailto:notifications@github.com>> wrote: Step 0: Investigate what community has already done in this direction. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#3938 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABKtU6yCcF6NMr86BCDwWwGyrN8663Lhks5sGCMrgaJpZM4OAEtS>.

[pdurbin]

The most recent work on Dockerizing Dataverse was done by @joelmarkanderson and @telnoratti in https://github.com/IQSS/dataverse-aws but I haven't heard from them in a bit. Hopefully they'll see this comment and let us know if they'd be willing to help us get up to speed with Dataverse on Docker and let us know the state of that repo.

I also tested bjonnh@f7a82c9 by @bjonnh a while back and it worked fine. This was an attempt to add a Dockerfile and related files directly to the main Dataverse repo. The goal was to have a place to run integration tests (API tests using REST Assured) on pull requests as they come in.

Finally, we can take a look at https://github.com/nds-org/ndslabs-dataverse/blob/master/dockerfiles/dataverse/Dockerfile by @craig-willis

If there are other Dataverse on Docker efforts out there, they aren't top of mind for me. Sorry if I'm forgetting anyone.

[pdurbin]

Yesterday @bjonnh and I chatted about how much work with thing this issue entails. Our guess is a full-time week for someone who already knows Docker:

bjonnh: pdurbin: saw your ping on github
pdurbin ah, this ping: #3938 (comment)
pdurbin bjonnh: I was wondering about docker build -t bjo/dataverse ... I assume we don't want "bjo" in there since it's you. :)
bjonnh: I still have some work to do on it
bjonnh: I see no problem giving it to something like dataverse/dataverse
bjonnh: well
bjonnh: I would like a docker compose that spins up multiple containers linked together
bjonnh: se we can separate solr and everything properly
bjonnh: that would help isolating issues
pdurbin bjonnh: ok. I think we might be attempting to estimate #3938 as soon as this afternoon. Do you have any idea how much work this would be?
bjonnh: in term of what?
bjonnh: I think it could be achieved in a full-time week (which I wish I would have), but don't quote me on that
bjonnh: pdurbin: ^
pdurbin bjonnh: ok, so a week for someone who actually knows Docker. Thanks!
bjonnh: yep

In other news, over at #3927 I'm asking developers who use Windows if they'd be interested in running Dataverse within Docker on their dev machines. In the context of a Windows dev environment, only one opinion on Docker has been given so far, which is "it could be a little difficult and it seems not necessary now."

[craig-willis]

We'd welcome an official set of Dataverse images for use in our system. Note that we also have separate Dockerfiles for Rserve, Solr, and TwoRavens. These are all implemented to fit into the NDS Labs Workbench, but could also work with docker compose.

https://github.com/nds-org/ndslabs-dataverse/tree/master/dockerfiles

It's worth noting that I hacked the Dataverse install process a fair amount to try to separate out the installation process, which could go into "docker build" from the configuration process needed during "docker run". This hasn't been revisited in 9+ months.

[pameyer]

@craig-willis I'm glad you saw this; the work that NDS Labs Workbench did on dockerizing Dataverse (and it's dependencies/components) has come up in some of these discussions.

There's been a little bit of work done with the Dataverse installer (more for non-interactive use than splitting components), but this installer might be getting reworked (#3937) : so any suggestions / comments about areas that could go smoother for this type of installation would be helpful.

[pameyer]

One thing that didn't occur to me to mention during initial discussions is that there may be a conflict between file storage (glassfish file storage directory), shared/persistent storage in docker, and running glassfish as root. May turn out not to be an issue, but something to check.

[pdurbin]

I just wanted to note for @craig-willis and others that I just read at nds-org/ndslabs-dataverse#8 that @moayadnajd is interested in running a version of Dataverse newer than 4.2.3: https://github.com/nds-org/ndslabs-dataverse/blob/9ddc9efa54185ffd69e25487159a09c4bb2e56bf/dockerfiles/dataverse/Dockerfile#L49 . I'm not sure that I'm qualified to work on that issue but maybe @bjonnh @donsizemore or @pameyer would know what to do... and I'm happy to learn! 😄

[moayadnajd]

@pdurbin thanks for mention that we using docker in production and its good for us because we are node.js and php developers and we don't know anything about java, glassfish server and tomcat so we was very happy when we saw the nds-org/ndslabs-dataverse but we had problems because we need to use handles and the version in that docker image not supporting handles i try to modify the dockerfile but i think there is a lot i miss because i don't how things work in dataverse

[pdurbin]

From @craig-willis via nds-org/ndslabs-dataverse#8 (comment)

By DDL I was referring to data definition language, aka SQL schema.

During my initial Dockerization of Dataverse, I had problems with
EclipseLink as it is configured. In the Docker environment, containers
can be easily brought up and down. The current persistence.xml defaults
to “create-tables” when the webapp is deployed. If a container is
restarted, resulting in redeployment of the webapp, startup fails during
table creation. I worked around this with the static DDL/schema which
is used to initialize the database during container startup if it's
never been run before, and setting eclipselink.ddl-generation to "none"
in persistence.xml.

There may be a better way to do this with EclipseLink, but I never found it.

I'm not sure what we should do in the Docker image. All I know is that we heavily depend on Glassfish creating our database tables. I assume we should be using https://flywaydb.org or similar to manage database migrations.

@moayadnajd I've been talking to @omaralsoudanii at nds-org/ndslabs-dataverse#8 (comment) and it sounds like Dataverse 4.7 is now available in Docker thanks to nds-org/ndslabs-dataverse@master...craig-willis:upgrade-4.7 by @craig-willis . For Handle support questions, please try one or more of the channels at http://guides.dataverse.org/en/4.7/installation/intro.html#getting-help ?

[moayadnajd]

@pdurbin and @craig-willis thanks for that we have it up and running without any issues but we will inform you if there any thanks a lot

[pdurbin]

Over at #4040 I'm trying to get the Dataverse Docker images working in OpenShift.

[pdurbin]

thanks for mention that we using docker in production

@moayadnajd I'm hacking away on Docker images for Dataverse in #4040 . Do you have any advice for me on what you like and don't like in Docker images? I don't have much experience with Docker! 😄

@omaralsoudanii I'd be happy to hear your thoughts on this as well!

[pdurbin]

Is anyone out there interested in testing a new Docker environment for working on Dataverse code? I just made pull request #4168 and you are welcome to try it out and give me feedback! Thanks!

[pdurbin]

After standup I spoke with @djbrooke and @landreev about both #4040 and #3938 (this issue) and I'm going to summarize the decisions made.

We're moving #4040 into the backlog and will continue to iterate on next steps in a future sprint. For details, please see #4040 (comment)

We'd like to put this issue through QA because it represents a toehold into the world of Docker for Dataverse developers. As a nice bonus, it's also a toehold into the world of Docker orchestratiion using Minishift, which is a developer-oriented build of OpenShift, which itself is a packaging of Kubernetes. Lots of fany new toys for Dataverse developers to play with!

@landreev may grab me for a demo before sending this issue to QA. The bulk of the pull request is documentation and config files.