Automate build instructions using Ansible #1213

samolisov · 2018-02-20T13:38:22Z

Ansible is the most popular open source automation tool on GitHub today
with more than a quarter million downloads per month.

The ansible playbook produces a docker image based on Ubuntu 16.4. The docker
image contains the Eclipse OpenJ9 virtual machine. The owner of the virtual
machine is the openj9 user (configurable) who is a member of the openj9 group
(configurable). A container created over the image runs as the openj9
non-privileged user.

The build process is broken down into the following phases:

create a docker container with the build environment
checkout the sources of ibmruntimes, omr and openj9
build openj9 from sources
archive the jdk
create a docker image and put the jdk inside
clean-up the build-environment

How to run the build:

$  ansible-playbook -i inventories/local/hosts openj9-docker-image.yml

Note: Ansible 2.4+ is required.

pshipton · 2018-02-20T14:03:21Z

Hi @samolisov. Thanks for the contribution. Before we can accept your contribution (which still needs to be reviewed) please follow the contribution guidelines https://github.com/eclipse/openj9/blob/master/CONTRIBUTING.md in the "Legal considerations" section in order to pass the ip-validation check. Until this check passes the contribution cannot be accepted.

All the files should be updated with the Eclipse & Apache license which you can copy from almost any other file, such as https://github.com/eclipse/openj9/blob/master/.travis.yml

samolisov · 2018-02-20T14:06:55Z

Hi @pshipton

I've added the headers (just squashing the commit). Unfortunately, the build doesn't work, it looks like fremarker is not available.

pshipton · 2018-02-20T14:09:54Z

The docker file (i.e. https://github.com/eclipse/openj9/blob/master/buildenv/docker/jdk8/x86_64/ubuntu16/Dockerfile) shows how to get freemarker. There are some instructions for specifying the location of freemarker here https://www.eclipse.org/openj9/oj9_build.html

samolisov · 2018-02-20T14:43:01Z

@pshipton Freemarker wasn't available not only for me but also for the Travis CI environment as well. Now everything is ok.

I've activated my ECA record and added the license headers to all files. Could anyone trigger the Travis CI build if it's required for the merge process?

pshipton · 2018-02-20T14:46:32Z

Please push a new commit to re-run the ip-validation check, we can't restart it.

samolisov · 2018-02-20T15:28:14Z

@pshipton Thank you for the advice.

All checks have passed

AdamBrousseau · 2018-02-20T18:59:55Z

buildenv/ansible/roles/openj9-build-environment/tasks/main.yml

+  tags:
+    - buildenv
+    - build
+- name: Install python to container if there isn''t one


nit pick isn''t -> isn't

Changed to name: Install python to the container if there is no one.

mstoodle · 2018-03-07T20:52:06Z

@samolisov if I'm not mistaken, you're not an IBM employee. In which case, you may not want to include IBM's copyright notice (top line in each file header) in the new files you're contributing. You can have your own copyright if you're submitting under your own name, otherwise your employer may wish to exert their copyright. I am not giving you legal advice, just want to make you aware you have options.

Also, I would like to suggest changing ibm:ibmuser to openj9:openj9 or openj9:openj9user since this project is OpenJ9 and not owned by IBM.

mstoodle · 2018-03-07T20:53:02Z

buildenv/ansible/inventories/local/hosts

+    platform: x86_64                # ppc64le, s390x, x86_64
+    target:
+      namespace: eclipse            
+      javahome: /opt/ibm/openj9


here's another thing that shouldn't be associated with IBM, please

mstoodle · 2018-03-07T20:54:38Z

buildenv/ansible/roles/openj9-image/files/Dockerfile-ubuntu

+  && apt-get update
+
+# Set the java home directory
+ARG JAVA_HOME=/opt/ibm/openj9


another ibm here

mstoodle · 2018-03-07T20:55:05Z

buildenv/ansible/roles/openj9-image/files/Dockerfile-ubuntu

+  && chown -f $USER:$GROUP $JAVA_HOME/sample || :
+  # small hach to ensure chown for a non-exist directory won't interrupt the build process.
+
+# Change to the IBM user.


and an unnecessary IBM here that should be removed :)

mstoodle

Please update all the ibm and IBM references (in the files as well as the commit message) and consider if you want to change the copyright statements you've put into these new files.

Ansible is the most popular open source automation tool on GitHub today with more than a quarter million downloads per month. The ansible playbook produces a docker image based on Ubuntu 16.4. The docker image contains the Eclipse OpenJ9 virtual machine. The owner of the virtual machine is the 'openj9' user (configurable) who is a member of the 'openj9' group (configurable). A container created over the image runs as the 'openj9' non-privileged user. The build process is broken down into the following phases: - create a docker container with the build environment - checkout the sources of ibmruntimes, omr and openj9 - build openj9 from sources - archive the jdk - create a docker image and put the jdk inside - clean-up the build-environment (the phase might be skipped) To start the build do: $ ansible-playbook -i inventories/local/hosts openj9-docker-image.yml Signed-off-by: Pavel Samolysov <samolisov@gmail.com>

samolisov · 2018-03-11T08:05:16Z

@mstoodle Thank you very much for your comments. Yes, I left IBM about 1.5 years ago and currently I'm not an IBM employee, so I've considered changing the copyright header to my name. Also, I've changed all IBM occurrences in the configuration and Dockerfile to openj9.

Unfortunately, currently (11.03.2018 10:00-11:00 MSK) the make all command doesn't work, it looks like there was a commit that has broken the build:

...
"Makefile:36: recipe for target 'ddr-scanner' failed", "Makefile:48: recipe for target 'lib' failed", 
"GNUmakefile:266: recipe for target 'ddr' failed", "GNUmakefile:32: recipe for target 'phase_omr' failed", 
"OpenJ9.gmk:322: recipe for target 'build-j9' failed"

samolisov · 2018-03-18T06:42:40Z

Copyright checks fail since I've change the copyright header for the new files. Does the PR have a chance to be merged? // cc @mstoodle

AdamBrousseau · 2018-03-18T12:04:06Z

Jenkins copyright check

AdamBrousseau · 2018-03-18T22:49:41Z

IBM was recently removed from the copyright check via #1446

mstoodle · 2018-03-19T13:32:50Z

@samolisov Are you still seeing the build problems you mentioned on March 11 ? I'm loathe to merge the PR if the scripts aren't quite working.

Also one other question: can you think of a way we could make an executable test out of these files, so that we can easily make sure they keep working once they get merged?

Finally, would you please update the ibms and ibmusers in the merge request? I saw that you fixed the commit message already (thanks!).

samolisov · 2018-03-20T06:25:39Z

@mstoodle I can't see the problem anymore:

$ docker image ls

REPOSITORY                                                          TAG                           IMAGE ID            CREATED            SIZE
eclipse/openj9-jdk8                    linux-x86_64-20180319202723   b8426d36e151        13 hours ago      447MB

$ docker run -it --name openj9-jdk8-new -v /home/ansible/hostdir:/opt/hostdir eclipse/openj9-jdk8:linux-x86_64-20180319202723
openj9@96e5788ea010:/opt/openj9$ whoami
openj9
openj9@96e5788ea010:/opt/openj9$ java -version
openjdk version "1.8.0_162-internal"
OpenJDK Runtime Environment (build 1.8.0_162-internal-_2018_03_19_15_12-b00)
Eclipse OpenJ9 VM (build master-8f002eb, JRE 1.8.0 Linux amd64-64 Compressed References 20180319_000000 (JIT enabled, AOT enabled)
OpenJ9   - 8f002eb
OMR      - e79d614
JCL      - 7580d49 based on jdk8u162-b12)
openj9@96e5788ea010:/opt/openj9$

About testing Ansible playbooks, please have a look at an amazing Ansible documentation: Check Mode (“Dry Run”). A brief description: this command will report what changes the playbook would have made rather than making them:

$ ansible-playbook -i inventories/local/hosts openj9-docker-image.yml --check --diff

But it works only if an Ansible module supports dry run else a task should just be ignored but, as I see, the docker_image module tries to build an image in the check mode and fails because there is no a build directory on the file system: the command for creating it was skipped.

Generally speaking, testing of Infrastructure as Code is a common problem in DevOps. If you don't run your scripts automatically and often, in some time they become absolutely unusable. The better way maybe is just to use this playbook on a CI platform to build and test Eclipse OpenJ9 docker images.

mstoodle · 2018-03-28T14:57:10Z

@Sorry for the delay @samolisov!

@AdamBrousseau are you also ok with this PR at this point?

AdamBrousseau

lftm

pshipton · 2018-03-28T16:18:13Z

@AdamBrousseau or @jdekonin are you able to run the build and confirm that it still works?

samolisov · 2018-04-25T11:10:24Z

Is the PR still actual for the project?

jdekonin · 2018-04-26T18:41:07Z

I haven't been able to get this to work, mind you I have very limited ansible experience. The error I am getting is ERROR! ERROR! inventories/local/hosts:24: Expected key=value host variable assignment, got: linux.

The existing build scripts used on jenkins via pipelines are how we use them in a larger across more than one platform at a time scale, but I am not opposed to different methods of generating a OpenJ9 binary, this is just a method that I personally am not familiar with.

mstoodle · 2018-05-11T16:24:23Z

I am no expert in this area, but I finally "broke down" and installed ansible into a docker container to try this out, and had the same experience as @jdekonin reported above (expecting key=value in the inventory file). For giggles, I also tried removing that -i option and got the following output:

 [WARNING]: provided hosts list is empty, only localhost is available

ERROR! no action detected in task

The error appears to have been in '/openj9/buildenv/ansible/roles/openj9-build-environment/tasks/main.yml': line 55, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

    - buildenv
- name: Start the building environment as a docker container
  ^ here

...which just looks like a file is not being interpreted correctly by something.

@samolisov we're probably making foolish noob mistakes, can you help to point us in the right direction? I was just following the direction in your commit message (i.e. I just ran ansible-playbook -i inventories/local/hosts openj9-docker-image.yml while sitting in the buildenv/ansible directory). Is there more to it than that?

samolisov · 2018-05-14T07:53:43Z

@mstoodle @jdekonin Thank you for the interest to the PR!

Could you check the ansible version, please? I tested all stuff on Ansible 2.4.2.0. The error you got with the inventory file seems to indicate that ansible can't interpret a yaml-formatted inventory file, a new in version 2.4 feature. This is my bad, unfortunately; I've added the minimum version info to the PR's description.

jdekonin · 2018-05-28T20:32:38Z

Editted: I am running docker on a mac to load ub16, I thought it was with ansible 2.4.2.0 but now I am doubting...trying again.

ymanton · 2018-05-31T13:44:34Z

Tried this, works fine on an Ubuntu 16.04 VM and creates what was expected. The only wrinkles are that you need a newer version of docker than is provided in the 16.04 repos, and you need docker-py, which I installed from pip.

fjeremic · 2018-05-31T14:18:32Z

Looks like a great way to automate things on our build farm as well. Looking forward to seeing this merged!

ymanton · 2018-07-05T19:53:47Z

Unless there are any objections I'm going to merge this tomorrow-ish since it has value and works as advertised.

ymanton · 2018-07-06T17:13:18Z

Merged. Thanks for the contribution.

pshipton added the comp:build label Feb 20, 2018

AdamBrousseau reviewed Feb 20, 2018

View reviewed changes

mstoodle reviewed Mar 7, 2018

View reviewed changes

mstoodle requested changes Mar 7, 2018

View reviewed changes

mstoodle approved these changes Mar 26, 2018

View reviewed changes

AdamBrousseau approved these changes Mar 28, 2018

View reviewed changes

ymanton merged commit 25f9acb into eclipse-openj9:master Jul 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automate build instructions using Ansible #1213

Automate build instructions using Ansible #1213

samolisov commented Feb 20, 2018 •

edited

Loading

pshipton commented Feb 20, 2018 •

edited

Loading

samolisov commented Feb 20, 2018

pshipton commented Feb 20, 2018

samolisov commented Feb 20, 2018

pshipton commented Feb 20, 2018

samolisov commented Feb 20, 2018

AdamBrousseau Feb 20, 2018

samolisov Feb 20, 2018 •

edited

Loading

mstoodle commented Mar 7, 2018

mstoodle Mar 7, 2018

mstoodle Mar 7, 2018

mstoodle Mar 7, 2018

mstoodle left a comment

samolisov commented Mar 11, 2018 •

edited

Loading

samolisov commented Mar 18, 2018

AdamBrousseau commented Mar 18, 2018

AdamBrousseau commented Mar 18, 2018

mstoodle commented Mar 19, 2018 •

edited

Loading

samolisov commented Mar 20, 2018 •

edited

Loading

mstoodle commented Mar 28, 2018

AdamBrousseau left a comment

pshipton commented Mar 28, 2018

samolisov commented Apr 25, 2018

jdekonin commented Apr 26, 2018

mstoodle commented May 11, 2018 •

edited

Loading

samolisov commented May 14, 2018

jdekonin commented May 28, 2018 •

edited

Loading

ymanton commented May 31, 2018

fjeremic commented May 31, 2018

ymanton commented Jul 5, 2018

ymanton commented Jul 6, 2018

Automate build instructions using Ansible #1213

Automate build instructions using Ansible #1213

Conversation

samolisov commented Feb 20, 2018 • edited Loading

pshipton commented Feb 20, 2018 • edited Loading

samolisov commented Feb 20, 2018

pshipton commented Feb 20, 2018

samolisov commented Feb 20, 2018

pshipton commented Feb 20, 2018

samolisov commented Feb 20, 2018

AdamBrousseau Feb 20, 2018

Choose a reason for hiding this comment

samolisov Feb 20, 2018 • edited Loading

Choose a reason for hiding this comment

mstoodle commented Mar 7, 2018

mstoodle Mar 7, 2018

Choose a reason for hiding this comment

mstoodle Mar 7, 2018

Choose a reason for hiding this comment

mstoodle Mar 7, 2018

Choose a reason for hiding this comment

mstoodle left a comment

Choose a reason for hiding this comment

samolisov commented Mar 11, 2018 • edited Loading

samolisov commented Mar 18, 2018

AdamBrousseau commented Mar 18, 2018

AdamBrousseau commented Mar 18, 2018

mstoodle commented Mar 19, 2018 • edited Loading

samolisov commented Mar 20, 2018 • edited Loading

mstoodle commented Mar 28, 2018

AdamBrousseau left a comment

Choose a reason for hiding this comment

pshipton commented Mar 28, 2018

samolisov commented Apr 25, 2018

jdekonin commented Apr 26, 2018

mstoodle commented May 11, 2018 • edited Loading

samolisov commented May 14, 2018

jdekonin commented May 28, 2018 • edited Loading

ymanton commented May 31, 2018

fjeremic commented May 31, 2018

ymanton commented Jul 5, 2018

ymanton commented Jul 6, 2018

samolisov commented Feb 20, 2018 •

edited

Loading

pshipton commented Feb 20, 2018 •

edited

Loading

samolisov Feb 20, 2018 •

edited

Loading

samolisov commented Mar 11, 2018 •

edited

Loading

mstoodle commented Mar 19, 2018 •

edited

Loading

samolisov commented Mar 20, 2018 •

edited

Loading

mstoodle commented May 11, 2018 •

edited

Loading

jdekonin commented May 28, 2018 •

edited

Loading