run-testenv job fails due to a timeout while waiting for netbird api key to be available

[kKdH]

The github action job run-testenv fails due to the following error:

thread 'main' panicked at .ci/docker/theo/src/core/docker.rs:156:13:
Timeout while waiting for netbird api key to be available.

First time observed here and here. But the job does not always fail. A successful run can be found here.

[reimarstier]

Observation:
Netbird management cannot communicate with Keycloak.
It tries to enumerate the users in keycloak but does not get the expected response.
The provisioning of netbird fails to authenticate and it cannot create an API token for the netbird service.

Error log in netbird management:

2024-01-18T15:34:59Z INFO management/cmd/management.go:250: running gRPC backward compatibility server: [::]:33073
2024-01-18T15:34:59Z INFO management/cmd/management.go:282: running HTTP server and gRPC server on the same port: [::]:80
2024-01-18T15:34:59Z WARN management/server/account.go:868: failed warming up cache due to error: unable to get http://keycloak/admin/realms/netbird/users/count?, statusCode 403
2024-01-18T15:35:02Z INFO management/server/account.go:1524: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2024-01-18T15:35:02Z ERRO management/server/http/middleware/auth_middleware.go:82: Error when validating JWT claims: unable to get http://keycloak/admin/realms/netbird/users/count?, statusCode 403
2024-01-18T15:35:02Z ERRO management/server/http/util/util.go:80: got a handler error: token invalid
2024-01-18T15:35:02Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2071539231: GET /api/users status 401

Conclusion: Netbird is expecting a response from keycloak but the users are not yet provisioned.

Similar issue: netbirdio/netbird#996

FIX:

• Wait explicitly for keycloak provisioning
• Improve error handling in THEO
• Netbird provision entrypoint shell script: Enumerate users in keycloak before starting netbird management service