Metrics POC

[skabashnyuk]

What does this PR do?

With CHE_METRICS_ENABLED env variable enable prometheus metrics endpoint
Metrics POC

ClassLoaderMetrics

# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes 11501.0
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total 0.0

JvmMemoryMetrics Record metrics that report utilization of various memory and buffer pools.

# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{area="nonheap",id="Code Cache",} 2.3396352E7
jvm_memory_committed_bytes{area="nonheap",id="Metaspace",} 6.9337088E7
jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space",} 8519680.0
jvm_memory_committed_bytes{area="heap",id="PS Eden Space",} 1.5204352E7
jvm_memory_committed_bytes{area="heap",id="PS Survivor Space",} 524288.0
jvm_memory_committed_bytes{area="heap",id="PS Old Gen",} 4.9283072E7
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="nonheap",id="Code Cache",} 2.2897152E7
jvm_memory_used_bytes{area="nonheap",id="Metaspace",} 6.7874864E7
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",} 8108328.0
jvm_memory_used_bytes{area="heap",id="PS Eden Space",} 1.322024E7
jvm_memory_used_bytes{area="heap",id="PS Survivor Space",} 484352.0
jvm_memory_used_bytes{area="heap",id="PS Old Gen",} 4.0086792E7
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="nonheap",id="Code Cache",} 2.5165824E8
jvm_memory_max_bytes{area="nonheap",id="Metaspace",} -1.0
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{area="heap",id="PS Eden Space",} 1.77733632E8
jvm_memory_max_bytes{area="heap",id="PS Survivor Space",} 524288.0
jvm_memory_max_bytes{area="heap",id="PS Old Gen",} 3.58088704E8

# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="direct",} 17.0
jvm_buffer_count_buffers{id="mapped",} 0.0
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="direct",} 515770.0
jvm_buffer_memory_used_bytes{id="mapped",} 0.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{id="direct",} 515770.0
jvm_buffer_total_capacity_bytes{id="mapped",} 0.0

JvmGcMetrics Record metrics that report a number of statistics related to garbage collection emanating from the MXBean and also adds information about GC causes.

# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total 5.75668224E8
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes 3.58088704E8
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes 3.9021816E7
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Allocation Failure",} 37.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Allocation Failure",} 0.444
jvm_gc_pause_seconds_count{action="end of minor GC",cause="GCLocker Initiated GC",} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="GCLocker Initiated GC",} 0.006
jvm_gc_pause_seconds_count{action="end of major GC",cause="Ergonomics",} 1.0
jvm_gc_pause_seconds_sum{action="end of major GC",cause="Ergonomics",} 0.136
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Allocation Failure",} 0.045
jvm_gc_pause_seconds_max{action="end of minor GC",cause="GCLocker Initiated GC",} 0.006
jvm_gc_pause_seconds_max{action="end of major GC",cause="Ergonomics",} 0.136
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total 1.0042528E7

JvmThreadMetrics

# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads 43.0
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 43.0
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{state="new",} 0.0
jvm_threads_states_threads{state="runnable",} 11.0
jvm_threads_states_threads{state="blocked",} 0.0
jvm_threads_states_threads{state="waiting",} 25.0
jvm_threads_states_threads{state="timed-waiting",} 7.0
jvm_threads_states_threads{state="terminated",} 0.0
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads 38.0

LogbackMetrics

# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{level="error",} 0.0
logback_events_total{level="warn",} 0.0
logback_events_total{level="info",} 0.0
logback_events_total{level="debug",} 0.0
logback_events_total{level="trace",} 0.0

FileDescriptorMetrics File descriptor metrics gathered by the JVM.

# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files 1048576.0
# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files 77.0

ProcessorMetrics Record metrics related to the CPU, gathered by the JVM..

# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage 0.002553191489361702
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count 4.0
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage 0.13787234042553193

UptimeMetrics

# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.542714699656E9
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds 96.138

FileStoresMeterBinder disk usage metrics

# HELP disk_free_bytes Unallocated space for file store
# TYPE disk_free_bytes gauge
disk_free_bytes{path="overlay",} 2.1418078208E10
disk_free_bytes{path="proc",} 0.0
disk_free_bytes{path="tmpfs",} 6.7108864E7
disk_free_bytes{path="devpts",} 0.0
disk_free_bytes{path="sysfs",} 0.0
disk_free_bytes{path="openrc",} 0.0
disk_free_bytes{path="cpuset",} 0.0
disk_free_bytes{path="cpu",} 0.0
disk_free_bytes{path="cpuacct",} 0.0
disk_free_bytes{path="blkio",} 0.0
disk_free_bytes{path="memory",} 0.0
disk_free_bytes{path="devices",} 0.0
disk_free_bytes{path="freezer",} 0.0
disk_free_bytes{path="net_cls",} 0.0
disk_free_bytes{path="perf_event",} 0.0
disk_free_bytes{path="net_prio",} 0.0
disk_free_bytes{path="hugetlb",} 0.0
disk_free_bytes{path="pids",} 0.0
disk_free_bytes{path="cgroup",} 0.0
disk_free_bytes{path="mqueue",} 0.0
disk_free_bytes{path="/dev/sda1",} 2.1418078208E10
disk_free_bytes{path="shm",} 6.7108864E7
# HELP disk_total_bytes Total space for file store
# TYPE disk_total_bytes gauge
disk_total_bytes{path="overlay",} 6.7371577344E10
disk_total_bytes{path="proc",} 0.0
disk_total_bytes{path="tmpfs",} 6.7108864E7
disk_total_bytes{path="devpts",} 0.0
disk_total_bytes{path="sysfs",} 0.0
disk_total_bytes{path="openrc",} 0.0
disk_total_bytes{path="cpuset",} 0.0
disk_total_bytes{path="cpu",} 0.0
disk_total_bytes{path="cpuacct",} 0.0
disk_total_bytes{path="blkio",} 0.0
disk_total_bytes{path="memory",} 0.0
disk_total_bytes{path="devices",} 0.0
disk_total_bytes{path="freezer",} 0.0
disk_total_bytes{path="net_cls",} 0.0
disk_total_bytes{path="perf_event",} 0.0
disk_total_bytes{path="net_prio",} 0.0
disk_total_bytes{path="hugetlb",} 0.0
disk_total_bytes{path="pids",} 0.0
disk_total_bytes{path="cgroup",} 0.0
disk_total_bytes{path="mqueue",} 0.0
disk_total_bytes{path="/dev/sda1",} 6.7371577344E10
disk_total_bytes{path="shm",} 6.7108864E7
# HELP disk_usable_bytes Usable space for file store
# TYPE disk_usable_bytes gauge
disk_usable_bytes{path="overlay",} 1.7965379584E10
disk_usable_bytes{path="proc",} 0.0
disk_usable_bytes{path="tmpfs",} 6.7108864E7
disk_usable_bytes{path="devpts",} 0.0
disk_usable_bytes{path="sysfs",} 0.0
disk_usable_bytes{path="openrc",} 0.0
disk_usable_bytes{path="cpuset",} 0.0
disk_usable_bytes{path="cpu",} 0.0
disk_usable_bytes{path="cpuacct",} 0.0
disk_usable_bytes{path="blkio",} 0.0
disk_usable_bytes{path="memory",} 0.0
disk_usable_bytes{path="devices",} 0.0
disk_usable_bytes{path="freezer",} 0.0
disk_usable_bytes{path="net_cls",} 0.0
disk_usable_bytes{path="perf_event",} 0.0
disk_usable_bytes{path="net_prio",} 0.0
disk_usable_bytes{path="hugetlb",} 0.0
disk_usable_bytes{path="pids",} 0.0
disk_usable_bytes{path="cgroup",} 0.0
disk_usable_bytes{path="mqueue",} 0.0
disk_usable_bytes{path="/dev/sda1",} 1.7965379584E10
disk_usable_bytes{path="shm",} 6.7108864E7

TomcatMetrics

# HELP tomcat_servlet_error_total
# TYPE tomcat_servlet_error_total counter
tomcat_servlet_error_total{name="default",} 0.0
tomcat_servlet_error_total{name="jsp",} 0.0
# HELP tomcat_sessions_active_max_sessions
# TYPE tomcat_sessions_active_max_sessions gauge
tomcat_sessions_active_max_sessions 11.0
# HELP tomcat_sessions_created_sessions_total
# TYPE tomcat_sessions_created_sessions_total counter
tomcat_sessions_created_sessions_total 11.0
# HELP tomcat_global_sent_bytes_total
# TYPE tomcat_global_sent_bytes_total counter
tomcat_global_sent_bytes_total{name="http-nio-8080",} 2321.0
# HELP tomcat_sessions_active_current_sessions
# TYPE tomcat_sessions_active_current_sessions gauge
tomcat_sessions_active_current_sessions 11.0
# HELP tomcat_global_request_max_seconds
# TYPE tomcat_global_request_max_seconds gauge
tomcat_global_request_max_seconds{name="http-nio-8080",} 0.748
# HELP tomcat_global_received_bytes_total
# TYPE tomcat_global_received_bytes_total counter
tomcat_global_received_bytes_total{name="http-nio-8080",} 0.0
# HELP tomcat_cache_hit_total
# TYPE tomcat_cache_hit_total counter
tomcat_cache_hit_total 0.0
# HELP tomcat_global_request_seconds
# TYPE tomcat_global_request_seconds summary
tomcat_global_request_seconds_count{name="http-nio-8080",} 11.0
tomcat_global_request_seconds_sum{name="http-nio-8080",} 0.798
# HELP tomcat_global_error_total
# TYPE tomcat_global_error_total counter
tomcat_global_error_total{name="http-nio-8080",} 0.0
# HELP tomcat_servlet_request_seconds
# TYPE tomcat_servlet_request_seconds summary
tomcat_servlet_request_seconds_count{name="default",} 0.0
tomcat_servlet_request_seconds_sum{name="default",} 0.0
tomcat_servlet_request_seconds_count{name="jsp",} 0.0
tomcat_servlet_request_seconds_sum{name="jsp",} 0.0
# HELP tomcat_sessions_alive_max_seconds
# TYPE tomcat_sessions_alive_max_seconds gauge
tomcat_sessions_alive_max_seconds 0.0
# HELP tomcat_sessions_rejected_sessions_total
# TYPE tomcat_sessions_rejected_sessions_total counter
tomcat_sessions_rejected_sessions_total 0.0
# HELP tomcat_cache_access_total
# TYPE tomcat_cache_access_total counter
tomcat_cache_access_total 0.0
# HELP tomcat_threads_config_max_threads
# TYPE tomcat_threads_config_max_threads gauge
tomcat_threads_config_max_threads{name="http-nio-8080",} 300.0
# HELP tomcat_sessions_expired_sessions_total
# TYPE tomcat_sessions_expired_sessions_total counter
tomcat_sessions_expired_sessions_total 0.0
# HELP tomcat_servlet_request_max_seconds
# TYPE tomcat_servlet_request_max_seconds gauge
tomcat_servlet_request_max_seconds{name="default",} 0.0
tomcat_servlet_request_max_seconds{name="jsp",} 0.0
# HELP tomcat_threads_current_threads
# TYPE tomcat_threads_current_threads gauge
tomcat_threads_current_threads{name="http-nio-8080",} 10.0
# HELP tomcat_threads_busy_threads
# TYPE tomcat_threads_busy_threads gauge
tomcat_threads_busy_threads{name="http-nio-8080",} 0.0

What issues does this PR fix or reference?

Exposing metrics via HTTP for Prometheus consumption #11384
Introduce metrics dependencies eclipse-che/che-parent#94

Release Notes

n/a

Docs PR

n/a

[skabashnyuk]

ci-test

[skabashnyuk]

ci-build

[riuvshin]

Results of automated E2E tests of Eclipse Che Multiuser on OCP:
Build details
Test report
docker image: eclipseche/che-server:11990
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.

[riuvshin]

ci-build

[skabashnyuk]

ci-test

[garagatyi]

Looks good. Please, take a look at inlined comments.

[garagatyi]

@skabashnyuk are you going to add docs?

[skabashnyuk]

@skabashnyuk are you going to add docs?

I'm going to sync this topic with @slemeur @rkratky and @l0rd

[skabashnyuk]

ci-test

[riuvshin]

Results of automated E2E tests of Eclipse Che Multiuser on OCP:
Build details
Test report
docker image: eclipseche/che-server:11990
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.

[skabashnyuk]

ci-test

[riuvshin]

Results of automated E2E tests of Eclipse Che Multiuser on OCP:
Build details
Test report
docker image: eclipseche/che-server:11990
https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.

[skabashnyuk]

ci-build

[garagatyi]

@skabashnyuk I don't think that these docs will be created in a foreseeable future. I shouldn't approve the PR without docs - my bad. Hopefully, you'll be able to add them later or negotiate who can do that.

[skabashnyuk]

@garagatyi You should not feel yourself bad. Generally speaking all reasonable big features like this should have documentation. And It's fair to ask the same question about docs in my PR that I'm asking in yours. We both making POC code sometimes and the thing that is not regulated and I'm wary about is that it's not clear what amount of documentation is required for each particular POC. Speaking about this concrete feature you can expect that until next GA this documentation will be added.