osc exec, Docker images and Software Collections

[rhcarvalho]

tl;dr: Couldn't we somehow build the images in such a way that the appropriate Software Collections that it uses are enabled globally?

---

I noticed that the Docker images we use that leverage SCL have a shortcoming when trying to use what might be the main software binaries the image was meant for.

After setting up a working environment with the sample-app, one might want to run a command in a container. Running a simple command like date works just fine:

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- date
Thu Apr 30 09:09:54 UTC 2015

But to run ruby in a Ruby Image:

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- ruby --version
env: ruby: No such file or directory
Error: Error executing remote command: Error executing command in container: exit status 127

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- scl enable ruby200 "ruby --version"
ruby 2.0.0p353 (2013-11-22) [x86_64-linux]

The same for databases:

$ osc exec -p database-1-nsxjr -c ruby-helloworld-database -- mysql --version
env: mysql: No such file or directory
Error: Error executing remote command: Error executing command in container: exit status 127

$ osc exec -p database-1-nsxjr -c ruby-helloworld-database -- scl enable mysql55 "mysql --version"
mysql  Ver 14.14 Distrib 5.5.37, for Linux (x86_64) using readline 5.1

Couldn't we somehow build the images in such a way that the appropriate Software Collections that it uses are enabled globally?

As a consequence, this would enable us to simplify our .sti/* scripts, removing the need to enable SCL in each script, and, most importantly, would make it easier for users to customize their scripts, focusing on functionality rather than having to know and care about SCL.
For instance, a .sti/run script for a Python project could be as simple as:

#!/bin/bash
exec python app.py --db=$DATABASE_SERVICE_HOST:$DATABASE_SERVICE_PORT

---

As a reference, the behavior I'm looking for matches that of the official Ruby image from Docker Hub:

$ docker run --rm ruby:2.0.0 ruby --version
ruby 2.0.0p645 (2015-04-13 revision 50299) [x86_64-linux]

[ncdc]

osc exec uses the container's configured path, which I'm guessing in this case doesn't include the SCL locations. We'll have to think about ways to handle this.

Sent from my iPhone

On Apr 30, 2015, at 6:49 AM, Rodolfo Carvalho notifications@github.com wrote:

I noticed that the Docker images we use that leverage SCL have a shortcoming when trying to use what might be the main software binaries the image was meant for.

After setting up a working environment with the sample-app, one might want to run a command in a container. Running a simple command like date works just fine:

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- date
Thu Apr 30 09:09:54 UTC 2015
But to run ruby in a Ruby Image:

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- ruby --version
env: ruby: No such file or directory
Error: Error executing remote command: Error executing command in container: exit status 127

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- scl enable ruby200 "ruby --version"
ruby 2.0.0p353 (2013-11-22) [x86_64-linux]
The same for databases:

$ osc exec -p database-1-nsxjr -c ruby-helloworld-database -- mysql --version
env: mysql: No such file or directory
Error: Error executing remote command: Error executing command in container: exit status 127

$ osc exec -p database-1-nsxjr -c ruby-helloworld-database -- scl enable mysql55 "mysql --version"
mysql Ver 14.14 Distrib 5.5.37, for Linux (x86_64) using readline 5.1
Couldn't we somehow build the images in such a way that the appropriate Software Collections that it uses are enabled globally?

As a consequence, this would enable us to simplify our .sti/* scripts, removing the need to enable SCL in each script, and, most importantly, would make it easier for users to customize their scripts, focusing on functionality rather than having to know and care about SCL.
For instance, a .sti/run script for a Python project could be as simple as:

#!/bin/bash
exec python app.py --db=$DATABASE_SERVICE_HOST:$DATABASE_SERVICE_PORT
As a reference, the behavior I'm looking for matches that of the official Ruby image from Docker Hub:

$ docker run --rm ruby:2.0.0 ruby --version
ruby 2.0.0p645 (2015-04-13 revision 50299) [x86_64-linux]
—
Reply to this email directly or view it on GitHub.

[ncdc]

And I would also guess you'll see the same behavior with docker exec - can you try and report back?

Sent from my iPhone

On Apr 30, 2015, at 6:49 AM, Rodolfo Carvalho notifications@github.com wrote:

I noticed that the Docker images we use that leverage SCL have a shortcoming when trying to use what might be the main software binaries the image was meant for.

After setting up a working environment with the sample-app, one might want to run a command in a container. Running a simple command like date works just fine:

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- date
Thu Apr 30 09:09:54 UTC 2015
But to run ruby in a Ruby Image:

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- ruby --version
env: ruby: No such file or directory
Error: Error executing remote command: Error executing command in container: exit status 127

$ osc exec -p frontend-1-8zeyh -c ruby-helloworld -- scl enable ruby200 "ruby --version"
ruby 2.0.0p353 (2013-11-22) [x86_64-linux]
The same for databases:

$ osc exec -p database-1-nsxjr -c ruby-helloworld-database -- mysql --version
env: mysql: No such file or directory
Error: Error executing remote command: Error executing command in container: exit status 127

$ osc exec -p database-1-nsxjr -c ruby-helloworld-database -- scl enable mysql55 "mysql --version"
mysql Ver 14.14 Distrib 5.5.37, for Linux (x86_64) using readline 5.1
Couldn't we somehow build the images in such a way that the appropriate Software Collections that it uses are enabled globally?

As a consequence, this would enable us to simplify our .sti/* scripts, removing the need to enable SCL in each script, and, most importantly, would make it easier for users to customize their scripts, focusing on functionality rather than having to know and care about SCL.
For instance, a .sti/run script for a Python project could be as simple as:

#!/bin/bash
exec python app.py --db=$DATABASE_SERVICE_HOST:$DATABASE_SERVICE_PORT
As a reference, the behavior I'm looking for matches that of the official Ruby image from Docker Hub:

$ docker run --rm ruby:2.0.0 ruby --version
ruby 2.0.0p645 (2015-04-13 revision 50299) [x86_64-linux]
—
Reply to this email directly or view it on GitHub.

[rhcarvalho]

@ncdc yes yes, it's the very same thing with docker exec.

You're also right to the point with SCL. Enabling it changes the PATH, making new binaries available or take precedence over others from /bin, /usr/bin, etc.

I don't know what would be the best way to do that, but one possibility I see would be appending the path to the appropriate SCL binaries in /etc/profile.

[rhcarvalho]

sclorg/s2i-base-container#35 solved this partially. SCL now is automatically enabled as long as you wrap commands with bash -c:

$ oc exec -p "django-1-mcbkx" -- python --version                # cannot do this
Python 2.7.5
$ oc exec -p "django-1-mcbkx" -- bash -c "python --version"      # works!
Python 3.3.2

[ncoghlan]

In https://github.com/ncoghlan/rf-apitest/blob/master/Dockerfile (which installs the upstream SCL directly) I handled this for the main command by using ENTRYPOINT to always enable the SCL, and then CMD to run the desired operation.

This handles docker run (since that invokes ENTRYPOINT), but doesn't help with docker exec (since that bypasses both ENTRYPOINT and CMD)

[rhcarvalho]

@ncoghlan any reason you preferred that over the approach in sti-base?

In sti-base we set a few env vars that are picked up by bash to make it enable SCL by sourcing a file.
For instance, this is what the file looks like for Python 2.7:

unset BASH_ENV PROMPT_COMMAND ENV
source scl_source enable python27

[ncoghlan]

Why would I do that when ENTRYPOINT natively handles the use case? I'm OK with needing to enable the SCL manually under exec, since that lets me choose whether to access the system Python or the SCL one.

[rhcarvalho]

Still today we need to be aware of SCL and whether it is enabled or not. This issue has driven us to implement mechanisms to give a better user experience. Today, wrapping commands through bash or sh automatically enable the necessary collections for our images.

I don't think we can do much better than that unless there would be a way to enable SCL globally and permanently.

Before I close this, @hhorak would it be reasonable to have SCLs globally enabled as a feature request?

[hhorak]

There are ways (that are sometimes risky and thus not recommended officially) to enable Software Collections globally by having a file in /etc/profiles.d which does source scl_source enable <mycollection>. This enables the collection for all users outside of container.

However, as you can expect, this mechanism depends on the same apparatus as we use already -- shell needs to read /etc/profiles.d in the first place. So, this won't help us in containers.

Going one step back, the software collections is a concept that is build on this basic thing: we need to change environment variables by some logic (the variables content is not static). So, enabling a collection permanently in container means being able to change environment in the image by some logic. If we can make something like this to work:

ENV PATH=`scl enable rh-mariadb100 'env | grep ^PATH'`

... then we would do make it happen. However, something like that doesn't work in Dockerfile.

Anyway, the all above was to explain why I don't see anything we can do better right now. That doesn't mean somebody else can't find a better way though :)

[rhcarvalho]

Summarizing my brainstorming with @hhorak on IRC earlier today:

• the PATH manipulation is an incomplete solution, because there are many other environment variables that need to be set for things to work correctly. Forgetting some of them would show problems only under some specific circumstances, causing unexpected bugs. E.g.: LD_LIBRARY_PATH, X_SCLS.
• @hhorak: "I tried to submit RFE for docker to allow to set env variables based on some command output, but since this is a specific request not very usable generally, I don't expect it will be implemented: RFE: Environment variables defined by running a command in the image (not static set) moby/moby#14490"
• Another possibility was to "allow RHEL to install an SCL instead of something from the official repos" under the standard paths, /usr/bin, and so on. Apparently, this would be too complicated and require building the packages again.
• Idea: "what if we create a file /usr/bin/pip in the container that would do this: scl enable rh-python34 -- pip $@, we would have to have wrappers for all binaries that are located under /opt/.../usr/bin/"
  This doesn't work, as user installed libraries that include binaries would not know about the trick, e.g., pip install ipython -- the ipython executable would not have a wrapper to enable the SCL.
• Yet another idea: docker exec ... container-entrypoint pip ...
  This is essentially the same as having to call bash -c ..., and now people need to worry about the entrypoint script and what it does.

With the lack of much viable improvements in the short term, I'm closing this.