Revisions unable to become Ready due to InternalError

[mbrancato]

I realized this is likely related / duplicating #15753

What version of Knative?

v1.17.0
Deployed with the operator

Expected Behavior

Revisions should become ready

Actual Behavior

Revisions stuck in Deploying status.

Events:
  Type     Reason         Age    From                 Message
  ----     ------         ----   ----                 -------
  Warning  InternalError  5m35s  revision-controller  failed to update deployment "helloworld-go-00001-deployment": Operation cannot be fulfilled on deployments.apps "helloworld-go-00001-deployment": the object has been modified; please apply your changes to the latest version and try again

The only error I see in the logs are about the webhook:

{"severity":"ERROR","timestamp":"2025-02-28T03:35:22.931441695Z","logger":"webhook.ConfigMapWebhook","caller":"controller/controller.go:564","message":"Reconcile error","commit":"6265a8e","knative.dev/pod":"webhook-57ccdb4884-8jb72","knative.dev/traceid":"8f7c243f-6571-48a7-9355-1b0b48f0e296","knative.dev/key":"config.webhook.serving.knative.dev","duration":"31.958µs","error":"error retrieving webhook: validatingwebhookconfiguration.admissionregistration.k8s.io \"config.webhook.serving.knative.dev\" not found","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:564\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:541\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:489"}

Steps to Reproduce the Problem

With a fresh cluster (I'm using Rancher Desktop with K8s 1.31.4) do the following. If its not clear, you may want to pause between the operator deploy and CRD deploy to allow services to start.

helm repo add knative-operator https://knative.github.io/operator
helm install knative-operator knative-operator/knative-operator

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: knative-serving
---
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
EOF

cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: ghcr.io/knative/helloworld-go:latest
          env:
            - name: TARGET
              value: "Go Sample v1"
EOF

This results in the following Revision that fails to become Ready:

% kubectl describe rev
Name:         helloworld-go-00001
Namespace:    default
Labels:       serving.knative.dev/configuration=helloworld-go
              serving.knative.dev/configurationGeneration=1
              serving.knative.dev/configurationUID=d8b185cf-a7c2-47dd-a39d-f3c829bcd5f3
              serving.knative.dev/routingState=active
              serving.knative.dev/service=helloworld-go
              serving.knative.dev/serviceUID=5af748d0-bafb-432c-95d2-b14b4be97559
Annotations:  serving.knative.dev/routes: helloworld-go
              serving.knative.dev/routingStateModified: 2025-02-28T03:30:20Z
API Version:  serving.knative.dev/v1
Kind:         Revision
Metadata:
  Creation Timestamp:  2025-02-28T03:30:20Z
  Generation:          1
  Owner References:
    API Version:           serving.knative.dev/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Configuration
    Name:                  helloworld-go
    UID:                   d8b185cf-a7c2-47dd-a39d-f3c829bcd5f3
  Resource Version:        2111
  UID:                     5571678c-8375-4da8-8781-7d9c4cf15c5b
Spec:
  Container Concurrency:  0
  Containers:
    Env:
      Name:   TARGET
      Value:  Go Sample v1
    Image:    ghcr.io/knative/helloworld-go:latest
    Name:     user-container
    Readiness Probe:
      Success Threshold:  1
      Tcp Socket:
        Port:  0
    Resources:
  Timeout Seconds:  300
Status:
  Conditions:
    Last Transition Time:  2025-02-28T03:30:20Z
    Reason:                Deploying
    Severity:              Info
    Status:                Unknown
    Type:                  Active
    Last Transition Time:  2025-02-28T03:30:22Z
    Status:                True
    Type:                  ContainerHealthy
    Last Transition Time:  2025-02-28T03:30:20Z
    Reason:                Deploying
    Status:                Unknown
    Type:                  Ready
    Last Transition Time:  2025-02-28T03:30:20Z
    Reason:                Deploying
    Status:                Unknown
    Type:                  ResourcesAvailable
  Container Statuses:
    Image Digest:       ghcr.io/knative/helloworld-go@sha256:584dae6fbc79fbf2cd8f94168b75b200d16dd36121eea55845e775aa547b00c8
    Name:               user-container
  Observed Generation:  1
Events:
  Type     Reason         Age    From                 Message
  ----     ------         ----   ----                 -------
  Warning  InternalError  5m35s  revision-controller  failed to update deployment "helloworld-go-00001-deployment": Operation cannot be fulfilled on deployments.apps "helloworld-go-00001-deployment": the object has been modified; please apply your changes to the latest version and try again

[mbrancato]

I was able to resolve this by specifying the Kourier gateway for ingress. Although, I do find it to violate some unwritten rule that the default configuration should work in most cases, and configurations that require dependencies (such as Istio, which seems to be the case here) should require more configuration.

I used this CRD:

---
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  version: "1.16.1"
  ingress:
    kourier:
      enabled: true
  config:
    network:
      ingress-class: "kourier.ingress.networking.knative.dev"