Docker file use wrapping pre/post scripting for the JVM process

[gzoller]

Right now it appears the Docker file (bin/root) marshals all the JVM artifacts onto the classpath and fires up the JVM with the given class of the main program. The problem I'm having is that if the JVM dies, e.g. OutOfMemoryError the Docker container just stops--I have no visibility into why. It appears to be a clean termination.

What would be very useful is to use a bash "wrapper" script in /bin/root, which in turn fires off the JVM process. Then expose something in sbt that allows me to specify arbitrary bash code (or another script). That way the script could run pre-JVM launch and post-JVM termination scripts, passing the exit code of the JVM process. The post-JVM script could clean up, log something on a remote log, etc. I'd have some idea what went wrong before the container terminated.

For example in sbt there could be a dockerPreJvmScript and dockerPostJvmScript settings. Those file(s) if specified would wind up in the bin dir and referenced in the wrapper code in root.

[muuki88]

I'm not sure if this should be necessary. If you want to monitor your jvm process
there are other tools available that do this job (a lot better than any bash-script).

• JMX
• New Relic
• Data Dog
• Kamon.IO
• Appdynamics

[gzoller]

I'm not sure that handles my need. These packages do great at runtime monitoring. I'm focused on legs-up, sudden & immediate, hard-failure of the JVM (e.g. OutOfMemoryError). These things cannot be safely caught or handled in the JVM process, so the dying words of the JVM in these cases aren't typically logged except to stderr.

In the current case the docker run script, i.e. /bin/root in my case, simply stops and the container gracefully exits--clean but unhelpful. How would I be able to figure out what went wrong and why my container simply stopped? I need a way to access that "dying breath" and thought that a wrapper script could do that, like this perhaps:

ERROR=$( { <RUN_JVM_HERE> | sed s/Output/Useless/ > outfile; } 2>&1 )
./post.sh "$ERROR"

(A fancier version might not call post.sh if there was no error output, i.e. a non-panic stopping of the JVM.)

The post.sh script could send that output to New Relic or whatever distributed logging was in place, or even write it to a file if mapping container volumes somewhere on the host. Then I'd know what happened.

Is there another way to do this using current facilities?

[muuki88]

If your application throws an exception the docker container should fail as well.
Put this in the test-project-docker main file and build it.

object Main extends App {
  println("Hello world")

  throw new Exception("YOU FAILED!")
}

Run with

docker run docker-test:0.1.0  && echo ok || echo failed 

Output is

Hello world
Exception in thread "main" java.lang.Exception: YOU FAILED!
    at Main$delayedInit$body.apply(Main.scala:4)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:71)
    at scala.App$$anonfun$main$1.apply(App.scala:71)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
    at scala.App$class.main(App.scala:71)
    at Main$.main(Main.scala:1)
    at Main.main(Main.scala)
failed

You can start your docker container and log the output to where you want like this:

docker run docker-test:0.1.0  > output.log