FISH-5827 Stuck Thread count as MicroProfile Metric Gauge (Payara 5.x)

[jGauravGupta]

Description

This PR adds HealthCheck stats (Stuck thread count) to MicroProfile Metrics.

Payara 6 PR:

#5952

Testing

Testing Performed

asadmin start-domain
asadmin set-healthcheck-service-configuration --serviceName=stuck-thread --enabled=true
asadmin restart-domain

Open http://localhost:8080/metrics/vendor URL which so=houl contains the following result:

# TYPE vendor_system_cpu_load gauge
# HELP vendor_system_cpu_load Display the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values between 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
vendor_system_cpu_load 0.2421606522975489
# TYPE vendor_thread_stuck_count gauge
# HELP vendor_thread_stuck_count Displays the stuck thread count which is blocked, and can't return to the threadpool for a certain amount of time.
vendor_thread_stuck_count 1

Now disable health check stuck-thread:

asadmin set-healthcheck-service-configuration --serviceName=stuck-thread --enabled=false
asadmin restart-domain

Open http://localhost:8080/metrics/vendor URL which so=houl contains the following result:

# TYPE vendor_system_cpu_load gauge
# HELP vendor_system_cpu_load Display the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values between 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
vendor_system_cpu_load 0.2421606522975489

Testing Environment

Windows 11, JDK 11.0.16

[jGauravGupta]

jenkins test please

[Pandrex247]

See review on other PR, I believe there's a bug - it always shows at least 1 stuck thread

[jGauravGupta]

jenkins test please

[jGauravGupta]

jenkins test please

[Pandrex247]

Working better, but one issue - it's still running even when the health check service is disabled.
This seems counter intuitive to how it works elsewhere in the server.

[jGauravGupta]

Hi Andrew,

it's still running even when the health check service is disabled.

I checked and can't reproduce it.
When Stuck Thread disabled:

On accessing the http://localhost:8080/metrics/vendor:

# TYPE vendor_system_cpu_load gauge
# HELP vendor_system_cpu_load Display the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values between 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
vendor_system_cpu_load 0.388351853542798

When stuck thread enabled:

On accessing the http://localhost:8080/metrics/vendor:

# TYPE vendor_system_cpu_load gauge
# HELP vendor_system_cpu_load Display the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values between 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
vendor_system_cpu_load 0.16640233851087813
# TYPE vendor_thread_stuck_count gauge
# HELP vendor_thread_stuck_count Displays the stuck thread count which is blocked, and can't return to the threadpool for a certain amount of time.
vendor_thread_stuck_count 0
# TYPE vendor_thread_stuck_maxDuration gauge
# HELP vendor_thread_stuck_maxDuration Displays the maximum duration of stuck thread which is blocked, and can't return to the threadpool for a certain amount of time.
vendor_thread_stuck_maxDuration 0

[Pandrex247]

Sorry I should have been more explicit - I meant the "overall" health check service.

Stuck thread, hogging thread, CPU etc. are all sub-systems of the overall health check service.
Each of the subsystems can be on or off, and the overall health check service itself can also be on or off.

So the way it should work, to be consistent with how the healthcheck system itself works, is:

• HealthCheck enabled, stuck thread disabled - no metric
• HealthCheck enabled, stuck thread enabled - metric
• HealthCheck disabled, stuck thread disabled - no metric
• HealthCheck disabled, stuck thread enabled - no metric

[jGauravGupta]

jenkins test please