Duplicated checks in health check response

[MigotMateusz]

Describe the bug

When quarkus app is deployed on environment with multiple instances checks in health check response are repeated.
There are 3 instances and one instance returns expected behavior, second instance returns doubled response and third instance returns tripled instance.

Expected behavior

{
	"status": "UP",
	"checks": [
		{
			"name": "Kafka consumers health check",
			"status": "UP"
		},
		{
			"name": "Database connections health check",
			"status": "UP",
			"data": {
				"<default>": "UP"
			}
		}
	]
}

Actual behavior

{
	"status": "UP",
	"checks": [
		{
			"name": "Kafka consumers health check",
			"status": "UP"
		},
		{
			"name": "Kafka consumers health check",
			"status": "UP"
		},
		{
			"name": "Kafka consumers health check",
			"status": "UP"
		},
		{
			"name": "Database connections health check",
			"status": "UP",
			"data": {
				"<default>": "UP"
			}
		},
		{
			"name": "Database connections health check",
			"status": "UP",
			"data": {
				"<default>": "UP"
			}
		},
		{
			"name": "Database connections health check",
			"status": "UP",
			"data": {
				"<default>": "UP"
			}
		}
	]
}

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.14.3.Final

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @jmartisk (health), @xstefank (health)

[xstefank]

This is something strange. Can you give us some reproducer please?

I'm not able to reproduce this.

[roy-mdsol]

Im facing a similar bug. this happens only when I deploy the code onto the server. It doesn't happen when running in Dev mode.

[geoand]

We'll need a sample application in order to investigate this

[geoand]

Closing as we did not get a reproducer. Feel free to comment with any additional information if it becomes available

[bushrod]

This is happening to me too (also only on server, not locally). I logged the times and saw it is being called twice in rapid succession. It is duplicated even if the check name is identical with the differing timestamps in data. Maybe the checks could be de-dupified, if not fixed by not calling multiple times, by adding the check names to a set and converting that to array. I tried returning null from the custom check instead of up if the last call was < some milliseconds, but health reported down because of the null response.

{
    "status": "UP",
    "checks": [
        {
            "name": "2023-06-15T18:56:08.724432",
            "status": "UP"
        },
        {
            "name": "2023-06-15T18:56:08.724675",
            "status": "UP"
        }
    ]
}

[Kazeheki]

The issue only occurs when you set up liveness/readiness probes in your cluster.
As long as I didn't add them, the duplication didn't happen.
I created a sample project: https://github.com/Kazeheki/quarkus-smallrye-health-duplication-example

[geoand]

@Kazeheki thanks a lot for the reproducer!

Mind explaining what you did with that project and saw the duplicate checks?

[Kazeheki]

@geoand I deployed the application in a cluster and set liveness/readiness probes.
If you run the start-script in the reproducer, it will setup a local kubernetes cluster for you (using k3d). In the file k8s.conf.yaml you see how I defined the deployment and the liveness/readiness probes.
The application itself basically only consists of a liveness and readiness probe.

I did nothing more than deploying it in a cluster with those probes and opening the health request localhost:18080/q/health in a browser.
If you leave out the readiness/liveness (comment out l.27-44) probes in the kubernetes definitions and apply the 'k8s.conf.yaml' (kubectl apply -f k8s.conf.yaml) to the cluster again and reload the health page, then the duplication is gone.

[geoand]

Thanks!

Was there anyway you could reproduce this while running the Quarkus application locally?

[Kazeheki]

I didn't manage to see the duplication in any other case than the liveness/readiness probe usage.

I tried to setup a scheduler to call the health endpoints every second which didn't have the same outcome with duplication as cluster liveness/readiness probes.

I also tried to do calls from outside (e.g. watch -n1 curl localhost:8080/q/health/live) which also didn't cause the duplication.

I can only recreate the duplication in a kubernetes cluster with liveness/readiness probes.

[geoand]

Gotcha, thanks!

[geoand]

@xstefank it seems like SmallryeHealthReporter lacks the necessary concurrency controls. I opened smallrye/smallrye-health#486 to address this, but I would not concider the ultimate solution,
it's just a quick fix for this issue.
My recommendation would be to merge it, release SR Health to include it, update Quarkus and when you have the time,
you look into more thorough concurrency coverage.

@Kazeheki Thanks again for the reproducer, it was most helpful! I could reliably reproduce the issue using it and was able to test my fix against it.