[WP] Attempt to enable che-master monitoring #950

skabashnyuk · 2018-10-04T13:51:47Z

What does this PR do?

https://github.com/fabric8io-images/java/tree/master/images/centos/openjdk8/jdk
https://github.com/fabric8io/agent-bond
https://github.com/rhuss/jolokia
https://github.com/prometheus/jmx_exporter

What issues does this PR fix or reference?

How have you tested this PR?

centos-ci · 2018-10-04T13:51:48Z

Can one of the admins verify this patch?

skabashnyuk · 2018-10-04T13:52:49Z

[test]

garagatyi · 2018-10-04T13:53:15Z

[test]

centos-ci · 2018-10-04T14:17:36Z

@skabashnyuk The che server [build 630] has been successfully deployed.

http://rhche-prcheck-950.devtools-dev.ext.devshift.net

ibuziuk · 2018-10-05T07:19:04Z

@davidfestal could you please review the PR and provide some feedback / recommendations since have precious experience of setting up jolokia for che-tenant-maintainer
[1] https://github.com/fabric8-services/che-tenant-maintainer

davidfestal · 2018-10-05T07:25:00Z

Sure, I'm looking into it

davidfestal · 2018-10-05T07:44:19Z

Seems good to me.
Just 2 remarks:

There will probably be some actions from the SD team probably to make the Jolokia console available for users that don't have edit rights on the dsaas production namespace.
As soon as the Jolokia console is available for some developers, it might be useful to be able to configure the Jolokia access rights as well. I mean that JMX write actions (GC run, call of JMX methods, etc ...) might be allowed from prod-preview, but I assume they should be prevented in production.
Here is how I did this on the che-tenant-maintainer: fabric8-services/che-tenant-maintainer@5172b3d

garagatyi · 2018-10-05T07:54:05Z

dockerfiles/che-fabric8/Dockerfile

@@ -43,6 +43,22 @@ RUN mkdir -p      /etc/pcp /var/run/pcp /var/lib/pcp /var/log/pcp
 RUN chmod -R 0777 /etc/pcp /var/run/pcp /var/lib/pcp /var/log/pcp
 EXPOSE 44321

+
+# Agent bond including Jolokia and jmx_exporter


Since there is a centos image with JDK 8 and Agent Bond here I think we should use it instead of manually adding all of this into RhChe repo.

@ibuziuk @davidfestal @l0rd @amisevsk WDYT?

The question is: is there the equivalent for RHEL. We now manage 2 versions of the DockerFile.

BTW @skabashnyuk Do you plan to do the same changes in Dockerfile.rhel ?
That would be needed to have JMX on prod and prod-preview I assume, no ?

ur right, there is no rhel version. Maybe we should ask to provide one

garagatyi · 2018-10-05T07:59:11Z

Seems good to me.
Just 2 remarks:

There will probably be some actions from the SD team probably to make the Jolokia console available for users that don't have edit rights on the dsaas production namespace.

As soon as the Jolokia console is available for some developers, it might be useful to be able to configure the Jolokia access rights as well. I mean that JMX write actions (GC run, call of JMX methods, etc ...) might be allowed from prod-preview, but I assume they should be prevented in production.
Here is how I did this on the che-tenant-maintainer: fabric8-services/che-tenant-maintainer@5172b3d

cool!

davidfestal

Seem my comments
#950 (comment)
https://github.com/redhat-developer/rh-che/pull/950/files#r222920644

garagatyi · 2018-10-05T08:03:30Z

@davidfestal Is it possible to test these changes on dev-cluster at all? I'm not sure we can do that without merging and delivering to prod-preview

davidfestal · 2018-10-05T08:08:47Z

@garagatyi I don't see why we wouldn't be able to test this on dev-cluster. But I might be missing some point.
OTOH, to have access to the console and test on prod-preview, edit rights on the dsaas preview namespace will be required. The good point with dev-cluster is that all dev team member have edit rights in the namespace they deploy their Che server on (or the pr-check namespaces as well afaik).

garagatyi · 2018-10-05T08:15:23Z

@davidfestal I just don't understand how to test it on dev cluster. Could you elaborate? I would really love to understand that and the whole team would also benefit from sharing this info

davidfestal · 2018-10-05T08:23:10Z

When a server is deployed with Jolokia enabled, the Openshift console allows accessing the JMX web console through a link available on the POD configuration page:

You can test it (if you have edit rights on prod-preview) on the maintainer POD config page:

https://console.rh-idev.openshift.com/console/project/dsaas-preview/browse/pods/che-tenant-maintainer-80-2tdbx?tab=details

I don't see why it wouldn't work on dev-cluster the same way.

davidfestal · 2018-10-05T08:26:50Z

@skabashnyuk Shouldn't you enable the jolokia port in the RhChe deployment yaml file ? by adding these options to the Che server container ports:

          - containerPort: 8778
            name: jolokia

garagatyi

Test to check what is wrong with my ability to comment other comments in the PR

garagatyi · 2018-10-05T09:00:08Z

When a server is deployed with Jolokia enabled, the Openshift console allows accessing the JMX web console through a link available on the POD configuration page:

You can test it (if you have edit rights on prod-preview) on the maintainer POD config page:

https://console.rh-idev.openshift.com/console/project/dsaas-preview/browse/pods/che-tenant-maintainer-80-2tdbx?tab=details

I don't see why it wouldn't work on dev-cluster the same way.

Now it is clear. Thanks!

skabashnyuk · 2018-10-08T06:43:52Z

Sorry for the late response
Thanks, guys for your comments
@davidfestal I didn't know about what you already did the same for
https://github.com/fabric8-services/che-tenant-maintainer
I'll try to reuse as much as possible from its configuration in this pr.

The reason why I did this pr in downstream first is that in upstream Che is quite challenging to enable metrics.

riuvshin · 2018-10-22T12:09:37Z

what is the state of this PR?

skabashnyuk · 2018-10-22T12:22:09Z

Sorry for late response. Unfortunately, due to the shift in priorities, I won't be able to finalize this PR
@ibuziuk @davidfestal feel free to do with it whatever you think the best.

centos-ci · 2018-10-22T19:06:02Z

@skabashnyuk The che server [build 729] has been successfully deployed.

http://rhche-prcheck-950.devtools-dev.ext.devshift.net

davidfestal · 2018-10-22T19:15:12Z

I'll test it and finalize it as necessary

skabashnyuk · 2018-10-23T06:52:31Z

@davidfestal thank you.

skabashnyuk · 2018-12-07T09:33:32Z

will be done as part of #1111

Agent bond initial add

9280c73

benoitf added the status/code-review label Oct 4, 2018

ibuziuk requested a review from davidfestal October 5, 2018 07:01

garagatyi reviewed Oct 5, 2018

View reviewed changes

davidfestal suggested changes Oct 5, 2018

View reviewed changes

garagatyi reviewed Oct 5, 2018

View reviewed changes

skabashnyuk closed this Oct 22, 2018

skabashnyuk reopened this Oct 22, 2018

skabashnyuk mentioned this pull request Oct 23, 2018

RFE: Expose usage metrics via Che Server API eclipse-che/che#11242

Closed

3 tasks

ibuziuk mentioned this pull request Oct 23, 2018

Need to enable che server monitoring on prod / prod-preview via Jolokia & jmx_exporter #1004

Closed

skabashnyuk closed this Dec 7, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WP] Attempt to enable che-master monitoring #950

[WP] Attempt to enable che-master monitoring #950

skabashnyuk commented Oct 4, 2018

centos-ci commented Oct 4, 2018

skabashnyuk commented Oct 4, 2018

garagatyi commented Oct 4, 2018

centos-ci commented Oct 4, 2018

ibuziuk commented Oct 5, 2018

davidfestal commented Oct 5, 2018

davidfestal commented Oct 5, 2018

garagatyi Oct 5, 2018

garagatyi Oct 5, 2018

davidfestal Oct 5, 2018

davidfestal Oct 5, 2018

garagatyi Oct 5, 2018

garagatyi commented Oct 5, 2018

davidfestal left a comment

garagatyi commented Oct 5, 2018

davidfestal commented Oct 5, 2018

garagatyi commented Oct 5, 2018

davidfestal commented Oct 5, 2018

davidfestal commented Oct 5, 2018

garagatyi left a comment

garagatyi commented Oct 5, 2018

skabashnyuk commented Oct 8, 2018

riuvshin commented Oct 22, 2018

skabashnyuk commented Oct 22, 2018

centos-ci commented Oct 22, 2018

davidfestal commented Oct 22, 2018

skabashnyuk commented Oct 23, 2018

skabashnyuk commented Dec 7, 2018 •

edited

Loading

[WP] Attempt to enable che-master monitoring #950

[WP] Attempt to enable che-master monitoring #950

Conversation

skabashnyuk commented Oct 4, 2018

What does this PR do?

What issues does this PR fix or reference?

How have you tested this PR?

centos-ci commented Oct 4, 2018

skabashnyuk commented Oct 4, 2018

garagatyi commented Oct 4, 2018

centos-ci commented Oct 4, 2018

@skabashnyuk The che server [build 630] has been successfully deployed.

ibuziuk commented Oct 5, 2018

davidfestal commented Oct 5, 2018

davidfestal commented Oct 5, 2018

garagatyi Oct 5, 2018

Choose a reason for hiding this comment

garagatyi Oct 5, 2018

Choose a reason for hiding this comment

davidfestal Oct 5, 2018

Choose a reason for hiding this comment

davidfestal Oct 5, 2018

Choose a reason for hiding this comment

garagatyi Oct 5, 2018

Choose a reason for hiding this comment

garagatyi commented Oct 5, 2018

davidfestal left a comment

Choose a reason for hiding this comment

garagatyi commented Oct 5, 2018

davidfestal commented Oct 5, 2018

garagatyi commented Oct 5, 2018

davidfestal commented Oct 5, 2018

davidfestal commented Oct 5, 2018

garagatyi left a comment

Choose a reason for hiding this comment

garagatyi commented Oct 5, 2018

skabashnyuk commented Oct 8, 2018

riuvshin commented Oct 22, 2018

skabashnyuk commented Oct 22, 2018

centos-ci commented Oct 22, 2018

@skabashnyuk The che server [build 729] has been successfully deployed.

davidfestal commented Oct 22, 2018

skabashnyuk commented Oct 23, 2018

skabashnyuk commented Dec 7, 2018 • edited Loading

skabashnyuk commented Dec 7, 2018 •

edited

Loading