E2E tests for broker: TLS key pair rotation

[Leo6Leo]

Fixes #3373
Fixes #3376

Proposed Changes

The tests will be failing until all the PRs below get merged.

Bug Fix In progress

• Expose tls port for broker #3305
• Add the kafka-XXX.knative-eventing.svc as hostname #3306
• Modify the existing fileWatcher to make it more generic #3325
• Update the updateServerConfig function to pass in the cert value diectly #3337
• Use the new fileWatcher instead of secretWatcher  #3338 (Blocked by Modify the existing fileWatcher to make it more generic #3325)
• transportEncryptionFlags fetureFlag is empty #3376 (Solved in this PR)
• Change the name of the secretVolumePath #3303

Backporting

• Midstream: [release-v1.11] Cherry-pick E2E tests for broker: TLS key pair rotation #3287 openshift-knative/eventing-kafka-broker#869
• Upstream 1.11: [release-1.11] Cherry-pick E2E tests for broker: TLS key pair rotation #3287 #3387
• Upstream 1.12: Has been backported along with the release

Release Note

TLS Port Exposure: Enhanced security by exposing the TLS port for the broker (#3305).
Hostname Update: Added kafka-XXX.knative-eventing.svc as the default hostname (#3306).
File Watcher Enhancements: Revamped the existing fileWatcher for more generic use cases (#3325). This change also led to the replacement of the secretWatcher with the updated fileWatcher (#3338).
Server Config Update: Improved the updateServerConfig function to directly accept the certificate value (#3337).
Transport Encryption: Resolved the issue where transportEncryptionFlags featureFlag was returning empty (#3376).

Docs

[codecov]

Codecov Report

Merging #3287 (36c265b) into main (5b46ea2) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #3287      +/-   ##
============================================
- Coverage     61.54%   61.54%   -0.01%     
  Complexity      761      761              
============================================
  Files           181      181              
  Lines         12316    12322       +6     
  Branches        265      265              
============================================
+ Hits           7580     7583       +3     
- Misses         4140     4142       +2     
- Partials        596      597       +1     

Flag	Coverage Δ
java-unittests	70.84% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
control-plane/pkg/reconciler/broker/controller.go	75.00% <100.00%> (+1.31%)	⬆️
...ve/eventing/kafka/broker/core/metrics/Metrics.java	77.67% <ø> (ø)

... and 1 file with indirect coverage changes

[knative-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[Leo6Leo]

/cc @pierDipi
Currently I am stuck on a ProbeError bug. Here is an overview of what is going on.

Context

• The discussion on CNCF slack

The problem

I am trying to make the rekt test for kafka broker to validate the keypair rotation work as expected.

What I have done

The broker is installed here

1. Enable the strict transport mode by applying the yaml file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-features
  namespace: knative-eventing
  labels:
    knative.dev/config-propagation: original
    knative.dev/config-category: eventing
data:
  transport-encryption: Strict

2. Running the single rekt test

SYSTEM_NAMESPACE=knative-eventing BROKER_CLASS=Kafka go 
test -v -tags=e2e -count=1 -run TestBrokerTlsCARotation  -parallel=12 -timeout=30m ./test/e2e_new >> log.txt

3. While the rekt test is still running, get the brokers in the test namespace

k get broker -n test-*****

Get the respnse:

NAME||URL||AGE||READY||REASON
broker-XXXXX||http://xxxxx.xxxxx.xxx ||       63s  ||  True   

Found out that the broker is still has http address. So I manually enforce the https mode by doing the next step.

4. Manually change the code in control-plane/pkg/reconciler/broker/broker.go, make it https only. Removed the if statement. See the original code here

var addressableStatus duckv1.AddressStatus

		logger.Debug("haha Strict transport encryption enabled")
		caCerts, err := r.getCaCerts()
		if err != nil {
			return err
		}

		httpsAddress := receiver.HTTPSAddress(ingressHost, broker, caCerts)
		addressableStatus.Address = &httpsAddress
		addressableStatus.Addresses = []duckv1.Addressable{httpsAddress}
	

	proberAddressable := prober.NewAddressable{
		AddressStatus: &addressableStatus,
		ResourceKey: types.NamespacedName{
			Namespace: broker.GetNamespace(),
			Name:      broker.GetName(),
		},
	}

5. Re-apply all the config

./hack/run.sh deploy-infro
./hack/run.sh deploy

6. Running the single rekt test again and repeat the step 1 - 3

7. Have the new error shows up
   Have the

k get broker -n test-*****

get the response

NAME              URL   AGE   READY   REASON
broker-aygordap         12m   False   ProbeStatus

8. Check the kafka-controller log, and get the following error that might related to this problem

{"level":"error","ts":"2023-08-21T19:54:02.823Z","logger":"kafka-broker-controller","caller":"prober/prober.go:117","msg":"Failed probe","commit":"204c66b-dirty","knative.dev/pod":"kafka-controller-5d7d4f75d8-wggtp","scope":"prober","IP":"kafka-broker-ingress.knative-eventing.svc","address":"https://kafka-broker-ingress.knative-eventing.svc:8443/test-lbllinla/broker-bhsvlpkq","error":"Get \"https://kafka-broker-ingress.knative-eventing.svc:8443/test-lbllinla/broker-bhsvlpkq\": dial tcp 10.107.43.20:8443: connect: network is unreachable","stacktrace":"knative.dev/eventing-kafka-broker/control-plane/pkg/prober.probe\n\tknative.dev/eventing-kafka-broker/control-plane/pkg/prober/prober.go:117\nknative.dev/eventing-kafka-broker/control-plane/pkg/prober.(*asyncProber).Probe.func2\n\tknative.dev/eventing-kafka-broker/control-plane/pkg/prober/async_prober.go:149"}

9. Also try to describe the broker resource, get the following

Conditions:
    Last Transition Time:  2023-08-21T19:38:55Z
    Status:                Unknown
    Type:                  Addressable
    Last Transition Time:  2023-08-21T19:38:55Z
    Reason:                Config map knative-eventing/kafka-broker-brokers-triggers updated
    Status:                True
    Type:                  ConfigMapUpdated
    Last Transition Time:  2023-08-21T19:38:55Z
    Status:                True
    Type:                  ConfigParsed
    Last Transition Time:  2023-08-21T19:38:55Z
    Status:                True
    Type:                  DataPlaneAvailable
    Last Transition Time:  2023-08-21T19:39:39Z
    Message:               status: UnknownError
    Reason:                ProbeStatus
    Status:                False
    Type:                  ProbeSucceeded
    Last Transition Time:  2023-08-21T19:39:39Z
    Message:               status: UnknownError
    Reason:                ProbeStatus
    Status:                False
    Type:                  Ready
    Last Transition Time:  2023-08-21T19:38:55Z
    Reason:                Topic knative-broker-test-rfgyklnt-broker-aygordap created
    Status:                True
    Type:                  TopicReady

What next

As Calum suggested,

either the prober is being passed the wrong address, or the data plane isn't listening on that address + port

My next step would be using telepresence to debug, by setting the break point in the reconciler and the prober.go and async_prober.go, and see what happens there. But I haven't successfully used telepresence yet.

What help I would like to ask

1. Anything wrong or missing with my approach above?
2. Is my next step seems right? Should I pursuit on that to debug?
3. If 2 is right, might need some help on setting up the telepresence.

Thanks for your help! @pierDipi

[pierDipi]

@Leo6Leo I think, I'd investigate whether the receiver is creating the https server, do you have the logs of the data plane? in particular, kafka-broker-receiver do you see any logs related to the creation of the https server? If we don't have enough logs on the data plane, let's add them and iterate

[Cali0707]

If 2 is right, might need some help on setting up the telepresence.

@Leo6Leo there is an open issue for this, maybe want to pick it up? #3272

[Leo6Leo]

@Leo6Leo I think, I'd investigate whether the receiver is creating the https server, do you have the logs of the data plane? in particular, kafka-broker-receiver do you see any logs related to the creation of the https server? If we don't have enough logs on the data plane, let's add them and iterate

@pierDipi
With your suggestion, found the problem while creating the java receiver server. And that has been fixed in this PR. #3303

But the issue I mentioned above pertain after the previous problem is resolved. Set up the loggers and can see that https server is created successfully. However, the probeError still exist

[Leo6Leo]

/retest-required

[Leo6Leo]

/retest-required

[Leo6Leo]

/cc @Cali0707

[Leo6Leo]

/retest-required

[Leo6Leo]

/retest-required

[pierDipi]

/test reconciler-tests

[Leo6Leo]

/retest-required

[pierDipi]

/lgtm
/approve

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Leo6Leo, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Leo6Leo]

/retest-required

[pierDipi]

/cherry-pick release-1.11

[pierDipi]

@Leo6Leo can you take care of backporting to 1.11?

[knative-prow-robot]

@pierDipi: #3287 failed to apply on top of branch "release-1.11":

Applying: progress save
Applying: add the new cert rotation test
Applying: Update the rekt test
Applying: Format the files
Applying: workspace save
Using index info to reconstruct a base tree...
M	control-plane/pkg/reconciler/broker/broker.go
A	data-plane/benchmarks/src/main/java/dev/knative/eventing/kafka/broker/dispatcher/impl/filter/ExactFilterBenchmark.java
A	data-plane/benchmarks/src/main/java/dev/knative/eventing/kafka/broker/dispatcher/impl/filter/FilterBenchmark.java
M	data-plane/receiver/src/main/java/dev/knative/eventing/kafka/broker/receiver/impl/ReceiverVerticle.java
Falling back to patching base and 3-way merge...
Auto-merging data-plane/receiver/src/main/java/dev/knative/eventing/kafka/broker/receiver/impl/ReceiverVerticle.java
CONFLICT (content): Merge conflict in data-plane/receiver/src/main/java/dev/knative/eventing/kafka/broker/receiver/impl/ReceiverVerticle.java
Auto-merging data-plane/dispatcher/src/main/java/dev/knative/eventing/kafka/broker/dispatcher/impl/filter/subscriptionsapi/NotFilter.java
CONFLICT (content): Merge conflict in data-plane/dispatcher/src/main/java/dev/knative/eventing/kafka/broker/dispatcher/impl/filter/subscriptionsapi/NotFilter.java
CONFLICT (modify/delete): data-plane/benchmarks/src/main/java/dev/knative/eventing/kafka/broker/dispatcher/impl/filter/ExactFilterBenchmark.java deleted in HEAD and modified in workspace save. Version workspace save of data-plane/benchmarks/src/main/java/dev/knative/eventing/kafka/broker/dispatcher/impl/filter/ExactFilterBenchmark.java left in tree.
Auto-merging control-plane/pkg/reconciler/broker/broker.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0005 workspace save
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.