add Bash trap framework basics, refactored stack trace #8527

stevekuznetsov · 2016-04-15T15:44:47Z

This will begin the series of pulls that hopefully will bring in the trap framework.

@Miciah please (re-)review the Bash bits
@smarterclayton please review the last commit

smarterclayton · 2016-04-16T04:37:18Z

hack/build-cross.sh

@@ -29,7 +29,8 @@ fi

 # Build the primary client/server for all platforms
 OS_BUILD_PLATFORMS=("${platforms[@]}")
-os::build::build_binaries "${OS_CROSS_COMPILE_TARGETS[@]}"
+# Create a sub-shell so that we don't pollute the outer environment
+( os::build::build_binaries "${OS_CROSS_COMPILE_TARGETS[@]}" )


Why not make build_binaries itself run its work in a subshell, rather than propagating up and out?

My question was, also, if we have a work-around for the $GOBIN thing for cross-compiles, why not use that work-around for everything and never leak a changed $GOBIN, thereby not needing the subshell at all?

stevekuznetsov · 2016-04-18T15:43:10Z

@smarterclayton addressed comments.

New stack trace:

$ hack/build-go.sh 
++ Building go targets for linux/amd64: cmd/openshift cmd/oc
# github.com/openshift/origin/pkg/cmd/util
pkg/cmd/util/env.go:16: syntax error: unexpected Int, expecting (
pkg/cmd/util/env.go:25: Env redeclared in this block
    previous declaration at pkg/cmd/util/env.go:16
[ERROR] PID 19547: hack/common.sh:311: `GOOS=${platform%/*} GOARCH=${platform##*/} go install "${goflags[@]:+${goflags[@]}}" -ldflags "${version_ldflags}" "${nonstatics[@]}"` exited with status 2.
[INFO]      Stack Trace: 
[INFO]        1: hack/common.sh:311: `GOOS=${platform%/*} GOARCH=${platform##*/} go install "${goflags[@]:+${goflags[@]}}" -ldflags "${version_ldflags}" "${nonstatics[@]}"`
[INFO]        2: hack/common.sh:255: os::build::internal::build_binaries
[INFO]        3: hack/build-go.sh:36: os::build::build_binaries
[INFO]   Exiting with code 2.
[ERROR] PID 19521: hack/common.sh:253: `( os::build::internal::build_binaries "$@" )` exited with status 2.
[INFO]      Stack Trace: 
[INFO]        1: hack/common.sh:253: `( os::build::internal::build_binaries "$@" )`
[INFO]        2: hack/build-go.sh:36: os::build::build_binaries
[INFO]   Exiting with code 2.

Old stack trace:

$ hack/build-go.sh 
++ Building go targets for linux/amd64: cmd/openshift cmd/oc
# github.com/openshift/origin/pkg/cmd/util
pkg/cmd/util/env.go:16: syntax error: unexpected Int, expecting (
pkg/cmd/util/env.go:25: Env redeclared in this block
    previous declaration at pkg/cmd/util/env.go:16
!!! Error in hack/../hack/common.sh:301
    'GOOS=${platform%/*} GOARCH=${platform##*/} go install "${goflags[@]:+${goflags[@]}}" -ldflags "${version_ldflags}" "${nonstatics[@]}"' exited with status 2
Call stack:
    1: hack/../hack/common.sh:301 os::build::build_binaries(...)
    2: hack/build-go.sh:32 main(...)
Exiting with status 1
!!! Error in hack/../hack/common.sh:253
    '( os::build::setup_env; local version_ldflags; version_ldflags=$(os::build::ldflags); local goflags; eval "goflags=(${OS_GOFLAGS:-})"; local arg; for arg in "$@";
do
    if [[ "${arg}" == -* ]]; then
        goflags+=("${arg}");
    fi;
done; os::build::export_targets "$@"; local -a nonstatics=(); local -a tests=(); for binary in "${binaries[@]}";
do
    if [[ "${binary}" =~ ".test"$ ]]; then
        tests+=($binary);
    else
        nonstatics+=($binary);
    fi;
done; local host_platform=$(os::build::host_platform); local platform; for platform in "${platforms[@]}";
do
    echo "++ Building go targets for ${platform}:" "${targets[@]}"; mkdir -p "${OS_OUTPUT_BINPATH}/${platform}"; if [[ $platform == $host_platform ]]; then
        export GOBIN="${OS_OUTPUT_BINPATH}/${platform}";
    else
        unset GOBIN;
    fi; if [[ ${#nonstatics[@]} -gt 0 ]]; then
        GOOS=${platform%/*} GOARCH=${platform##*/} go install "${goflags[@]:+${goflags[@]}}" -ldflags "${version_ldflags}" "${nonstatics[@]}"; if [[ $platform != $host_platform ]]; then
            local platform_src="/${platform//\//_}"; mv "${OS_TARGET_BIN}/${platform_src}/"* "${OS_OUTPUT_BINPATH}/${platform}/";
        fi;
    fi; for test in "${tests[@]:+${tests[@]}}";
    do
        local outfile="${OS_OUTPUT_BINPATH}/${platform}/$(basename ${test})"; GOOS=${platform%/*} GOARCH=${platform##*/} go test -c -o "${outfile}" "${goflags[@]:+${goflags[@]}}" -ldflags "${version_ldflags}" "$(dirname ${test})";
    done;
done )' exited with status 1
Call stack:
    1: hack/../hack/common.sh:253 os::build::build_binaries(...)
    2: hack/build-go.sh:32 main(...)
Exiting with status 1

stevekuznetsov · 2016-04-18T15:56:45Z

[test]me

stevekuznetsov · 2016-04-18T17:22:35Z

Flake on #7429

re[test]

smarterclayton · 2016-04-18T19:42:28Z

Conformance flake is #8491

On Mon, Apr 18, 2016 at 2:25 PM, OpenShift Bot notifications@github.com
wrote:

continuous-integration/openshift-jenkins/test FAILURE (
https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/3067/)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#8527 (comment)

stevekuznetsov · 2016-04-20T12:33:43Z

hack/lib/log/stacktrace.sh

+        local bash_source="$( os::util::repository_relative_path "${BASH_SOURCE[$i]}" )"
+        if [[ -z "${preamble_finished-}" ]]; then
+            preamble_finished=true
+            os::log::error "PID ${BASHPID:-$$}: ${bash_source}:${BASH_LINENO[$i-1]}: \`${last_command}\` exited with status ${return_code}." >&2


@Miciah I understand ${BASHPID:-$$} isn't technically a loss-less default, but I'm not particularly interested in making this nice for people using 12-year-old-Bash, so instead of a clear indication that each stacktrace is for a failed process, they'll just always see the parent PID.

stevekuznetsov · 2016-04-20T14:25:23Z

integration flake: #8573

re[test]

stevekuznetsov · 2016-04-27T12:09:57Z

@Miciah were you able to take a look? If not, @eparis do you mind doing a review for this?

eparis · 2016-04-27T12:30:22Z

hack/lib/log/stacktrace.sh

+    local i
+    for (( i = stack_begin_index; i < ${#BASH_SOURCE[@]}; i++ )); do
+        local bash_source="$( os::util::repository_relative_path "${BASH_SOURCE[$i]}" )"
+        if [[ -z "${preamble_finished-}" ]]; then


missing : ? (it doesn't actually matter I guess, but it is more consistent)

Miciah · 2016-04-27T21:01:07Z

hack/common.sh

+  ( os::build::internal::build_binaries "$@" )  
+}
+
+# Build binaries targets specified. Should always be run in a sub-shell so we don't leak GOBIN


Seems like "binaries targets" here and on line 261 should be "binary targets" instead.

Miciah · 2016-04-27T21:20:48Z

Nothing uses os::util::trap::init, nothing uses os::util::trap::init_exit except for os::util::trap::init, and nothing sets START_TIME. I take it users will come in future commits?

Miciah · 2016-04-27T21:24:32Z

hack/test-tools.sh

+source "${OS_ROOT}/hack/lib/util/trap.sh"
+source "${OS_ROOT}/hack/lib/log/stacktrace.sh"
+
+trap os::test::junit::reconcile_output EXIT


This looks like a fix that is not directly related to the other changes, or am I mistaken?

Yes, good eyes - I just caught this while I was fixing preambles.

Miciah · 2016-04-27T21:25:21Z

Reviewed, commented on some minor things. Sorry for the delay!

stevekuznetsov · 2016-06-15T18:50:47Z

AWS Infra flake:

The instance never became "ready" in AWS.

re[test]

stevekuznetsov · 2016-06-15T20:37:16Z

re[test]

This changes the prior implementation of the stacktrace to use the new stacktrace implementation as well as the newly-implemented Bash trap framework for the ERR signal. In order to make the stacktrace output more useful, especially for someone attempting to re-run the commands used, we evaluate all of the variables in the $BASH_COMMAND as best we can. We will not be able to ever correctly evaluate positional variables ($1, $2, $@) or variables created in the local scopes of the trap handler or in the stacktrace function such as $return_code, $last_command, or $errexit_set. For this reason, we print both $BASH_COMMAND and our attempt to substitute variables. Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

By changing the amount of code that is literally contained in the sub-shell `()` braces, we allow the stacktrace that is generated to contain much less text on any one line, so that it is read-able and useful. Furthermore, by naming the positional variables we pass around we allow the new stack trace code to correctly evaluate them and show us what went wrong, exactly. Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

stevekuznetsov · 2016-06-15T21:52:44Z

@smarterclayton this is ready
On Jun 15, 2016 5:50 PM, "OpenShift Bot" notifications@github.com wrote:

continuous-integration/openshift-jenkins/test SUCCESS (
https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/4941/)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#8527 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AG_ScmgzbUOB93vrblkzslv4RJoVrbNtks5qMHOtgaJpZM4IIccN
.

stevekuznetsov · 2016-06-16T11:53:53Z

Well, there was a green run before. Not sure what kicked this one.

e2e flake:

FAILURE after 0.309s: test/end-to-end/core.sh:351: executing 'docker pull gcr.io/google_containers/pause' expecting success: the command returned the wrong error code
Standard output from the command:
Using default tag: latest
Trying to pull repository gcr.io/google_containers/pause ... 

Standard error from the command:
name invalid: Requested repository does not match bearer token resource: google_containers/pause

test-cmd flake:

FAILURE after 0.214s: test/cmd/router.sh:32: executing 'oadm router --dry-run --host-network=false --stats-port=1937 -o yaml' expecting failure and text 'hostPort: 1937': the output content test failed
Standard output from the command:
...
          - containerPort: 1937
            hostPort: 1937
...

Standard error from the command:
error: router could not be created; service account "router" is not allowed to access host ports on nodes, grant access with oadm policy add-scc-to-user hostnetwork -z router

re[test]

stevekuznetsov · 2016-06-16T14:39:00Z

Looks like a new flake in conformance and integration:

curl: (7) Failed connect to copr.fedorainfracloud.org:443; Connection refused

#9379

re[test]

stevekuznetsov · 2016-06-16T19:40:52Z

Extended failures:

yum metalink Infrastructure Flake on Jenkins when building openshift/node image: [Errno -1] repomd.xml does not match metalink for epel #8571
extended flake: build without output image building from templates should create an image from ".../test-docker-no-outputname.json" docker template without an output image reference defined #9364

test-cmd:

FAILURE after 0.415s: test/cmd/admin.sh:308: executing 'oadm registry --dry-run' expecting failure and text 'does not exist': the output content test failed
There was no output from the command.
Standard error from the command:
error: error getting client: Get https://127.0.0.1:28443/api: dial tcp 127.0.0.1:28443: getsockopt: connection refused

@smarterclayton you seen this before? ^^

re[test]

stevekuznetsov · 2016-06-16T21:42:06Z

@mfojtik @smarterclayton this looks like a rebase issue?

I0616 16:05:52.554723   19712 meta.go:46] Calling Accessor on non-internal object: *extensions.DaemonSetList
panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *extensions.DaemonSet

goroutine 5893 [running]:
panic(0x35dca60, 0xc82c065e80)
    /usr/lib/golang/src/runtime/panic.go:481 +0x3e6
k8s.io/kubernetes/pkg/controller/daemon.NewDaemonSetsController.func5(0x3611500, 0xc8274fe840)
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/daemon/controller.go:172 +0x18e
k8s.io/kubernetes/pkg/controller/framework.ResourceEventHandlerFuncs.OnDelete(0xc8250a00a0, 0xc8250a00b0, 0xc8250a00c0, 0x3611500, 0xc8274fe840)
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:184 +0x3a
k8s.io/kubernetes/pkg/controller/framework.(*ResourceEventHandlerFuncs).OnDelete(0xc82785aea0, 0x3611500, 0xc8274fe840)
    <autogenerated>:25 +0xb5
k8s.io/kubernetes/pkg/controller/framework.NewInformer.func1(0x36116c0, 0xc8274feca0, 0x0, 0x0)
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:254 +0x56d
k8s.io/kubernetes/pkg/controller/framework.(*Controller).processLoop(0xc8271d90a0)
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:128 +0x6f
k8s.io/kubernetes/pkg/controller/framework.(*Controller).(k8s.io/kubernetes/pkg/controller/framework.processLoop)-fm()
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x20
k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc8280fff78)
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:66 +0x4b
k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc8280fff78, 0x3b9aca00, 0x0, 0xc82005ae40)
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:67 +0x68
k8s.io/kubernetes/pkg/util/wait.Until(0xc8280fff78, 0x3b9aca00, 0xc82005ae40)
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47 +0x3e
k8s.io/kubernetes/pkg/controller/framework.(*Controller).Run(0xc8271d90a0, 0xc82005ae40)
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x1c2
created by k8s.io/kubernetes/pkg/controller/daemon.(*DaemonSetsController).Run
    /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/daemon/controller.go:224 +0xa6

mfojtik · 2016-06-16T21:44:01Z

it seems so, label it as post rebase, I will have a look tomorrow.

Sent from my phone

On 16 Jun 2016, at 23:42, Steve Kuznetsov notifications@github.com wrote:

@mfojtik https://github.com/mfojtik @smarterclayton
https://github.com/smarterclayton this looks like a rebase issue?

I0616 16:05:52.554723 19712 meta.go:46] Calling Accessor on
non-internal object: *extensions.DaemonSetList
panic: interface conversion: interface {} is
cache.DeletedFinalStateUnknown, not *extensions.DaemonSet

goroutine 5893 [running]:
panic(0x35dca60, 0xc82c065e80)
/usr/lib/golang/src/runtime/panic.go:481
+0x3e6k8s.io/kubernetes/pkg/controller/daemon.NewDaemonSetsController.func5(0x3611500,
0xc8274fe840)
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/daemon/controller.go:172
+0x18ek8s.io/kubernetes/pkg/controller/framework.ResourceEventHandlerFuncs.OnDelete(0xc8250a00a0,
0xc8250a00b0, 0xc8250a00c0, 0x3611500, 0xc8274fe840)
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:184
+0x3ak8s.io/kubernetes/pkg/controller/framework.(_ResourceEventHandlerFuncs).OnDelete(0xc82785aea0,
0x3611500, 0xc8274fe840)
:25
+0xb5k8s.io/kubernetes/pkg/controller/framework.NewInformer.func1(0x36116c0,
0xc8274feca0, 0x0, 0x0)
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:254
+0x56dk8s.io/kubernetes/pkg/controller/framework.(_Controller).processLoop(0xc8271d90a0)
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:128
+0x6fk8s.io/kubernetes/pkg/controller/framework.(_Controller).(k8s.io/kubernetes/pkg/controller/framework.processLoop)-fm()
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103
+0x20k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc8280fff78)
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:66
+0x4bk8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc8280fff78,
0x3b9aca00, 0x0, 0xc82005ae40)
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:67
+0x68k8s.io/kubernetes/pkg/util/wait.Until(0xc8280fff78, 0x3b9aca00,
0xc82005ae40)
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47
+0x3ek8s.io/kubernetes/pkg/controller/framework.(_Controller).Run(0xc8271d90a0,
0xc82005ae40)
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103
+0x1c2
created by k8s.io/kubernetes/pkg/controller/daemon.(*DaemonSetsController).Run
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/daemon/controller.go:224
+0xa6

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#8527 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AACsaMRifV5pIprQ26l1PfW3r2PNOulNks5qMcMvgaJpZM4IIccN
.

stevekuznetsov · 2016-06-20T19:08:52Z

re[test]

stevekuznetsov · 2016-06-20T20:33:19Z

flake 2x #8571

re[test]

stevekuznetsov · 2016-06-21T00:21:08Z

Hit #8571 again

re[test]

stevekuznetsov · 2016-06-21T17:39:46Z

hit #9457
re[test]

stevekuznetsov · 2016-06-21T19:24:39Z

hit #5448

re[test]

openshift-bot · 2016-06-21T19:25:31Z

Evaluated for origin test up to 62e2e0a

openshift-bot · 2016-06-21T20:43:12Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/5194/)

smarterclayton · 2016-06-21T23:29:36Z

LGTM [merge]

openshift-bot · 2016-06-21T23:30:26Z

Evaluated for origin merge up to 62e2e0a

openshift-bot · 2016-06-22T00:50:24Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/5194/) (Image: devenv-rhel7_4437)

stevekuznetsov · 2016-06-22T12:45:05Z

hit #5448
@smarterclayton please re-tag

stevekuznetsov · 2016-06-22T19:57:52Z

I never thought the day would come.
🎆 🎊 🎉 🎆 🎊 🎉 🎆 🎊 🎉 🎆 🎊 🎉

mfojtik · 2016-06-23T08:17:47Z

@stevekuznetsov I know that feeling, congrats!

rhcarvalho · 2016-07-13T15:12:01Z

Who knows, maybe one day it will become https://github.com/niieani/bash-oo-framework#error-handling-with-exceptions-and-throw...

stevekuznetsov · 2016-07-13T17:18:22Z

@rhcarvalho good lord ...

smarterclayton reviewed Apr 16, 2016
View reviewed changes

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 18, 2016

stevekuznetsov force-pushed the skuznets/stacktrace branch from 8407af7 to 71b84d1 Compare April 18, 2016 15:42

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 18, 2016

stevekuznetsov added kind/enhancement area/tests labels Apr 19, 2016

stevekuznetsov force-pushed the skuznets/stacktrace branch from 71b84d1 to 9af7f9f Compare April 20, 2016 12:20

stevekuznetsov reviewed Apr 20, 2016
View reviewed changes

eparis reviewed Apr 27, 2016
View reviewed changes

stevekuznetsov force-pushed the skuznets/stacktrace branch from 9af7f9f to c96500e Compare April 27, 2016 13:00

Miciah reviewed Apr 27, 2016
View reviewed changes

stevekuznetsov force-pushed the skuznets/stacktrace branch from 997c1e2 to 835a39f Compare June 15, 2016 17:13

stevekuznetsov force-pushed the skuznets/stacktrace branch from 835a39f to 830a7f7 Compare June 15, 2016 20:37

stevekuznetsov added 2 commits June 15, 2016 17:06

stevekuznetsov force-pushed the skuznets/stacktrace branch from 830a7f7 to 62e2e0a Compare June 15, 2016 21:06

openshift-bot merged commit 96e291c into openshift:master Jun 22, 2016

stevekuznetsov mentioned this pull request Aug 4, 2016

call stack in bash is hard to look at and overwhelming #6325

Closed

add Bash trap framework basics, refactored stack trace #8527

add Bash trap framework basics, refactored stack trace #8527

Conversation

stevekuznetsov commented Apr 15, 2016

smarterclayton Apr 16, 2016

Choose a reason for hiding this comment

stevekuznetsov Apr 18, 2016

Choose a reason for hiding this comment

stevekuznetsov commented Apr 18, 2016

stevekuznetsov commented Apr 18, 2016

stevekuznetsov commented Apr 18, 2016

smarterclayton commented Apr 18, 2016

stevekuznetsov Apr 20, 2016

Choose a reason for hiding this comment

stevekuznetsov commented Apr 20, 2016

stevekuznetsov commented Apr 27, 2016

eparis Apr 27, 2016 • edited Loading

Choose a reason for hiding this comment

Miciah Apr 27, 2016

Choose a reason for hiding this comment

Miciah commented Apr 27, 2016

Miciah Apr 27, 2016

Choose a reason for hiding this comment

stevekuznetsov Apr 28, 2016

Choose a reason for hiding this comment

Miciah commented Apr 27, 2016

stevekuznetsov commented Jun 15, 2016

stevekuznetsov commented Jun 15, 2016

stevekuznetsov commented Jun 15, 2016

stevekuznetsov commented Jun 16, 2016 • edited Loading

stevekuznetsov commented Jun 16, 2016

stevekuznetsov commented Jun 16, 2016

stevekuznetsov commented Jun 16, 2016

mfojtik commented Jun 16, 2016

stevekuznetsov commented Jun 20, 2016

stevekuznetsov commented Jun 20, 2016

stevekuznetsov commented Jun 21, 2016

stevekuznetsov commented Jun 21, 2016

stevekuznetsov commented Jun 21, 2016

openshift-bot commented Jun 21, 2016

openshift-bot commented Jun 21, 2016

smarterclayton commented Jun 21, 2016

openshift-bot commented Jun 21, 2016

openshift-bot commented Jun 22, 2016 • edited Loading

stevekuznetsov commented Jun 22, 2016

stevekuznetsov commented Jun 22, 2016

mfojtik commented Jun 23, 2016 • edited Loading

rhcarvalho commented Jul 13, 2016

stevekuznetsov commented Jul 13, 2016

eparis Apr 27, 2016 •

edited

Loading

stevekuznetsov commented Jun 16, 2016 •

edited

Loading

openshift-bot commented Jun 22, 2016 •

edited

Loading

mfojtik commented Jun 23, 2016 •

edited

Loading