UAA initialization does not complete

[kei-yamazaki]

Describe the bug

Initialization process in the UAA initialization container build-uaa-truststore remains runnning and does not complete.

To Reproduce*

Steps to reproduce the behavior:

1. kapp deploy -a cf -f <(ytt -f config -f ${TMP_DIR}/cf-values.yml)

Details of UAA pods are as follows.

% kubectl describe pods -n cf-system uaa-b74fcf46b-p77pg
Name:         uaa-b74fcf46b-p77pg
Namespace:    cf-system
Priority:     0
Node:         kind-control-plane/172.21.0.2
Start Time:   Tue, 29 Mar 2022 10:27:17 +0900
Labels:       app=uaa-deployment
              app.kubernetes.io/component=authorization_server
              app.kubernetes.io/instance=uaa-standalone
              app.kubernetes.io/managed-by=kubectl
              app.kubernetes.io/name=uaa
              app.kubernetes.io/part-of=uaa
              app.kubernetes.io/version=74.12.0
              cloudfoundry.org/istio_version=1.11.4
              kapp.k14s.io/app=1648517138640864000
              kapp.k14s.io/association=v1.e8707e23a13a5ea9490e70e9123fd989
              pod-template-hash=b74fcf46b
              security.istio.io/tlsMode=istio
              service.istio.io/canonical-name=uaa
              service.istio.io/canonical-revision=74.12.0
Annotations:  prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15020
              prometheus.io/scrape: true
              sidecar.istio.io/proxyCPU: 0m
              sidecar.istio.io/proxyMemory: 0M
              sidecar.istio.io/status:
                {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-...
Status:       Pending
IP:           10.244.0.19
IPs:
  IP:           10.244.0.19
Controlled By:  ReplicaSet/uaa-b74fcf46b
Init Containers:
  build-uaa-truststore:
    Container ID:  containerd://c9e99149f0e79a2d52c0ba9c414fadd5ff85450cd4754701a20869dbe9db99b5
    Image:         cloudfoundry/uaa@sha256:2ad93078807501f7ac3cbcaa414c5871a7f09054c6470a718b0e021780d16ee8
    Image ID:      docker.io/cloudfoundry/uaa@sha256:2ad93078807501f7ac3cbcaa414c5871a7f09054c6470a718b0e021780d16ee8
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      find_keytool_or_fail_fast() {
        local has_keytool
        ls "${JAVA_HOME}"/bin/keytool
        has_keytool=$?

        if [ ${has_keytool} -ne 0 ]; then
          exit ${has_keytool}
        fi
      }

      import_cert() {
        local pemfile="${1}"
        local alias="${2}"

        echo "Adding ${pemfile} to truststore"

        # Have to use cat instead of -file
        # because keytool won't understand all of the filenames!
        cat "${pemfile}" | "${JAVA_HOME}"/bin/keytool \
          -noprompt \
          -import \
          -trustcacerts \
          -alias "${alias}" \
          -keystore "${TRUSTSTORE_FILE}" \
          -storepass "${TRUSTSTORE_PASSWORD}"
      }

      get_alias() {
        local pemfile="${1}"
        basename "${pemfile}" .pem
      }

      add_ca_certs() {
        local has_ca_certs
        ls ${SECRETS_DIR}/ca_certs/*.pem
        has_ca_certs=$?

        if [ ${has_ca_certs} -eq 0 ]; then
          for cert in ${SECRETS_DIR}/ca_certs/*.pem; do
            import_cert "${cert}" "$(get_alias $cert)"
          done
        fi
      }

      add_system_certs() {
        for cert in $OS_CERTS_DIR/*.pem; do
          import_cert "${cert}" "$(get_alias $cert)"
        done
      }

      main() {
        find_keytool_or_fail_fast
        add_ca_certs
        add_system_certs
      }

      main

    State:          Running
      Started:      Tue, 29 Mar 2022 10:28:18 +0900
    Ready:          False
    Restart Count:  0
    Environment:
      SECRETS_DIR:          /etc/secrets
      TRUSTSTORE_FILE:      /etc/truststore/uaa.pkcs12.truststore
      TRUSTSTORE_PASSWORD:  changeit
      JAVA_HOME:            /layers/paketo-buildpacks_bellsoft-liberica/jre
      OS_CERTS_DIR:         /etc/ssl/certs
    Mounts:
      /etc/secrets/ca_certs from ca-certs-files (ro)
      /etc/truststore from truststore-file (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from uaa-token-pgmxt (ro)
  istio-init:
    Container ID:
    Image:         index.docker.io/istio/proxyv2:1.11.4
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      *
      -d
      15090,15021,15020
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        0
      memory:     0
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from uaa-token-pgmxt (ro)
Containers:
  istio-proxy:
    Container ID:
    Image:         index.docker.io/istio/proxyv2:1.11.4
    Image ID:
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
      --concurrency
      2
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      0
      memory:   0
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                    third-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      uaa-b74fcf46b-p77pg (v1:metadata.name)
      POD_NAMESPACE:                 cf-system (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      PROXY_CONFIG:                  {"holdApplicationUntilProxyStarts":true}

      ISTIO_META_POD_PORTS:          [
                                         {"name":"http-uaa","containerPort":8080,"protocol":"TCP"}
                                         ,{"name":"metrics-uaa","containerPort":9102,"protocol":"TCP"}
                                     ]
      ISTIO_META_APP_CONTAINERS:     uaa,statsd-exporter
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_META_WORKLOAD_NAME:      uaa
      ISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/cf-system/deployments/uaa
      ISTIO_META_MESH_ID:            cluster.local
      TRUST_DOMAIN:                  cluster.local
      ISTIO_PROMETHEUS_ANNOTATIONS:  {"scrape":"true","path":"/metrics","port":"9102"}
      ISTIO_KUBE_APP_PROBERS:        {"/app-health/statsd-exporter/livez":{"httpGet":{"path":"/metrics","port":9102,"scheme":"HTTP"},"timeoutSeconds":1},"/app-health/statsd-exporter/readyz":{"httpGet":{"path":"/metrics","port":9102,"scheme":"HTTP"},"timeoutSeconds":1}}
    Mounts:
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from uaa-token-pgmxt (ro)
      /var/run/secrets/tokens from istio-token (rw)
  uaa:
    Container ID:
    Image:          cloudfoundry/uaa@sha256:2ad93078807501f7ac3cbcaa414c5871a7f09054c6470a718b0e021780d16ee8
    Image ID:
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      BPL_TOMCAT_ACCESS_LOGGING:  y
      JAVA_OPTS:                  -Dspring_profiles=postgresql -Djava.security.egd=file:/dev/./urandom -Dlogging.config=/etc/config/log4j2.properties -Dlog4j.configurationFile=/etc/config/log4j2.properties -DCLOUDFOUNDRY_CONFIG_PATH=/etc/config -DSECRETS_DIR=/etc/secrets -Djavax.net.ssl.trustStore=/etc/truststore/uaa.pkcs12.truststore -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=changeit -Dstatsd.enabled=true -Dservlet.session-store=database
    Mounts:
      /etc/config from uaa-config (rw)
      /etc/secrets/admin_client_credentials.yml from admin-client-credentials-file (ro,path="admin_client_credentials.yml")
      /etc/secrets/cf_admin_user_credentials.yml from cf-admin-user-credentials-file (ro,path="cf_admin_user_credentials.yml")
      /etc/secrets/cf_api_backup_metadata_generator_client_credentials.yml from cf-api-backup-metadata-generator-client-credentials-file (ro,path="client_credentials.yml")
      /etc/secrets/cf_api_controllers_client_credentials.yml from cf-api-controllers-client-credentials-file (ro,path="client_credentials.yml")
      /etc/secrets/cf_client_credentials.yml from cc-admin-client-credentials-file (ro,path="client_credentials.yml")
      /etc/secrets/cloud_controller_username_lookup_client_credentials.yml from cloud-controller-username-lookup-client-credentials-file (ro,path="client_credentials.yml")
      /etc/secrets/database_credentials.yml from database-credentials-file (ro,path="database_credentials.yml")
      /etc/secrets/encryption_keys.yml from encryption-keys-file (ro,path="encryption_keys.yml")
      /etc/secrets/jwt_policy_signing_keys.yml from jwt-policy-signing-keys-file (ro,path="jwt_policy_signing_keys.yml")
      /etc/secrets/saml_keys.yml from saml-keys-file (ro,path="saml_keys.yml")
      /etc/secrets/smtp_credentials.yml from smtp-credentials-file (ro,path="smtp_credentials.yml")
      /etc/truststore from truststore-file (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from uaa-token-pgmxt (ro)
  statsd-exporter:
    Container ID:
    Image:         cloudfoundry/statsd_exporter-cf-for-k8s@sha256:d5f14b2d08acdc86ac2691b052db58dbe3b1029fe5d16c086da7215f5a65ba7d
    Image ID:
    Port:          9102/TCP
    Host Port:     0/TCP
    Args:
      --statsd.listen-udp=:8125
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:15020/app-health/statsd-exporter/livez delay=0s timeout=1s period=5s #success=1 #failure=2
    Readiness:      http-get http://:15020/app-health/statsd-exporter/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from uaa-token-pgmxt (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  uaa-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      uaa-config-ver-1
    Optional:  false
  smtp-credentials-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-smtp-credentials
    Optional:    true
  database-credentials-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-database-credentials-ver-1
    Optional:    true
  admin-client-credentials-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-admin-client-credentials-ver-1
    Optional:    false
  jwt-policy-signing-keys-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-jwt-policy-signing-keys-ver-1
    Optional:    false
  ca-certs-files:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-ca-certs
    Optional:    true
  truststore-file:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  saml-keys-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-saml-keys-ver-1
    Optional:    false
  encryption-keys-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  encryption-keys-ver-1
    Optional:    false
  cf-admin-user-credentials-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-cf-admin-user-credentials-ver-1
    Optional:    false
  cc-admin-client-credentials-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-cf-client-secret-ver-1
    Optional:    false
  cloud-controller-username-lookup-client-credentials-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-cloud-controller-username-lookup-client-secret-ver-1
    Optional:    false
  cf-api-controllers-client-credentials-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-cf-api-controllers-client-secret-ver-1
    Optional:    false
  cf-api-backup-metadata-generator-client-credentials-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-cf-api-backup-metadata-generator-client-secret-ver-1
    Optional:    false
  uaa-token-pgmxt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  uaa-token-pgmxt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

Expected behavior

UAA pods to be ready.

Additional context

cf-for-k8s SHA

v5.4.3

Deploy instructions

$ ./hack/generate-values.sh -d vcap.me > ${TMP_DIR}/cf-values.yml
$ cat << EOF >> ${TMP_DIR}/cf-values.yml
app_registry:
  hostname: https://index.docker.io/v1/
  repository_prefix: "<my_username>"
  username: "<my_username>"
  password: "<my_password>"

add_metrics_server_components: true
enable_automount_service_account_token: true
load_balancer:
  enable: false
metrics_server_prefer_internal_kubelet_address: true
remove_resource_requirements: true
EOF

$ kapp deploy -a cf -f <(ytt -f config -f ${TMP_DIR}/cf-values.yml)

Cluster information

% kind version
kind v0.12.0 go1.17.8 darwin/arm64

CLI versions

% ytt --version
ytt version 0.40.1

% kapp --version
kapp version 0.46.0

% kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:51:05Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-03-07T09:39:29Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/arm64"}
WARNING: version difference between client (1.23) and server (1.20) exceeds the supported minor version skew of +/-1

% cf version
cf version 7.4.0+e55633fed.2021-11-15

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/181721836

The labels on this github issue will be updated when the story is started.