katib-db-manager grpc binary not found in 0.16.0

[NohaIhab]

Description

channel: latest/edge

The grpc_health_probe binary was removed from katib-db-manager container in upstream changes from v0.15.0 to v0.16.0-rc, see PR.
Charmed katib-db-manager relies on this binary to do the Pebble health check.
With the absence of the grpc binary, the health check cannot be performed and is considered to fail by Pebble, causing the service to constantly restart.

Logs:

2023-08-31T09:55:55.385Z [pebble] Service "katib-db-manager" starting: ./katib-db-manager
2023-08-31T09:55:55.395Z [katib-db-manager] I0831 09:55:55.395346      14 db.go:32] Using MySQL
2023-08-31T09:55:56.390Z [pebble] GET /v1/changes/1/wait?timeout=4.000s 1.009228471s 200
2023-08-31T09:55:56.685Z [pebble] GET /v1/plan?format=yaml 263.776µs 200
2023-08-31T09:56:04.409Z [katib-db-manager] I0831 09:56:04.409069      14 init.go:27] Initializing v1beta1 DB schema
2023-08-31T09:56:04.445Z [katib-db-manager] I0831 09:56:04.445069      14 main.go:113] Start Katib manager: 0.0.0.0:6789
2023-08-31T09:56:55.374Z [pebble] Check "katib-db-manager-up" failure 1 (threshold 5): fork/exec /bin/grpc_health_probe: no such file or directory
2023-08-31T09:57:55.372Z [pebble] Check "katib-db-manager-up" failure 2 (threshold 5): fork/exec /bin/grpc_health_probe: no such file or directory
2023-08-31T09:58:55.373Z [pebble] Check "katib-db-manager-up" failure 3 (threshold 5): fork/exec /bin/grpc_health_probe: no such file or directory
2023-08-31T09:59:55.372Z [pebble] Check "katib-db-manager-up" failure 4 (threshold 5): fork/exec /bin/grpc_health_probe: no such file or directory

Solution

Add grpc binary to the katib rock in https://github.com/canonical/katib-rocks/tree/main/katib-db-manager, and integrate rock with katib-db-manager charm.

As a temporary fix, the pebble health check is to be disabled in the code with a comment linking to this issue, and re-enabled once the rock is integrated.

[i-chvets]

Formatted the issue a bit.

[NohaIhab]

Temporarily, the health checks can be removed until we add the grpc binary to the rock and integrate the rock into the katib-db-manager charm.

[i-chvets]

Temporarily, the health checks can be removed until we add the grpc binary to the rock and integrate the rock into the katib-db-manager charm.

@NohaIhab Can we add this under Solution as well? Basically described proper solution as you did, with may be a link to the ROCK in question. And then temporary fix until ROCK is integrated.