FISH-5827 Stuck Thread count as MicroProfile Metric Gauge

[jGauravGupta]

Description

This PR adds HealthCheck stats (Stuck thread count) to MicroProfile Metrics.

Important Info

StuckThreadsHealthCheck required by MetricsServiceImpl.
MetricsServiceImpl required by FaultToleranceServiceImpl and MicroProfileMetricsChecker.
All of these services started at @RunLevel(StartupRunLevel.VAL).
The enable status of HealthCheck can not be checked during registration of Metrics metadata which is by default registered and during response writer operation, health check status is checked.

Testing

Testing Performed

asadmin start-domain
asadmin set-healthcheck-service-configuration --serviceName=stuck-thread --enabled=true
asadmin restart-domain

Open http://localhost:8080/metrics/vendor URL which so=houl contains the following result:

# TYPE vendor_system_cpu_load gauge
# HELP vendor_system_cpu_load Display the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values between 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
vendor_system_cpu_load 0.2421606522975489
# TYPE vendor_thread_stuck_count gauge
# HELP vendor_thread_stuck_count Displays the stuck thread count which is blocked, and can't return to the threadpool for a certain amount of time.
vendor_thread_stuck_count 1

Now disable health check stuck-thread:

asadmin set-healthcheck-service-configuration --serviceName=stuck-thread --enabled=false
asadmin restart-domain

Open http://localhost:8080/metrics/vendor URL which so=houl contains the following result:

# TYPE vendor_system_cpu_load gauge
# HELP vendor_system_cpu_load Display the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values between 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
vendor_system_cpu_load 0.2421606522975489

Testing Environment

Windows 11, JDK 11.0.16

[Pandrex247]

Is there a reason it seems to always say there is a stuck thread?

[jGauravGupta]

Changing the base branch to Payara6-tck as JavaEE8 Samples requires CDI 4.0, see payara/patched-src-javaee8-samples@5956e84#diff-d26e10d72530f9acdf8b358d1aa078c602ed4aa38dea26ca966117cb1857e3a2

[jGauravGupta]

jenkins test please

[jGauravGupta]

Hi @Pandrex247,

At the start of the request to http://localhost:8080/metrics/vendor, the Stuck thread count is incremented by 1, and end of the request decremented as the metric writer processing is done in between it always shows at least 1.

(org.glassfish.grizzly.threadpool.DefaultWorkerThread) Thread[http-thread-pool::http-listener-1(4),5,main]

[Pandrex247]

I don't understand.
If I turn on the log notifier there are no listed stuck threads - I don't see why a call to this endpoint has a discrepancy, it seems like a bug

[Pandrex247]

Yeah I think this is a bug.
Setting the stuck thread threshold to 8 minutes and poking the endpoint still has it show 1 stuck thread.

[Pandrex247]

We're closing the Payara6-tck branch, changing base.

[jGauravGupta]

jenkins test please

[jGauravGupta]

jenkins test please