imc dispatcher metrics show duplicate entries with event_scheme=""

[maschmid]

Describe the bug

imc dispatcher shows duplicate metrics for inmemorychannel_dispatcher_event_count , once with event_scheme="http" and once with event_scheme="" . IIUC, the event_scheme should always be either "http" or "https", so the empty values are always incorrect.

$ curl http://127.0.0.1:9090/metrics 2>/dev/null | grep inmemorychannel_dispatcher_event_count | grep heartbeat
inmemorychannel_dispatcher_event_count{container_name="dispatcher",event_scheme="",event_type="dev.knative.eventing.samples.heartbeat",namespace_name="foobar",response_code="202",response_code_class="2xx",unique_name="imc-dispatcher-6fdfb6c6b6-4ghjq20d5202b49c875bf2309e76baa8d241f"} 59
inmemorychannel_dispatcher_event_count{container_name="dispatcher",event_scheme="http",event_type="dev.knative.eventing.samples.heartbeat",namespace_name="foobar",response_code="202",response_code_class="2xx",unique_name="imc-dispatcher-6fdfb6c6b6-4ghjq20d5202b49c875bf2309e76baa8d241f"} 59

This causes prometheus queries like sum(rate(inmemorychannel_dispatcher_event_count{namespace_name="foobar"}[1m])) to show double of the actual expected event counts.

Expected behavior
There should not be metrics with event_scheme="", if they have already been reported as event_scheme="http"

To Reproduce
deploy a heartbeats example container with a known event rate to an InMemoryChannel (see
https://gist.github.com/maschmid/5ddc22d7703f05e7d656dda7309132af for an example) , watch the imc dispatcher metrics.

Knative release version
1.12

Additional context
Deployed on OpenShift via Serverless Operator CI builds of 1.33 version from https://github.com/openshift-knative/serverless-operator current main branch

[maschmid]

The problem has been probably introduced by #7581 , with this bit https://github.com/knative/eventing/pull/7581/files#diff-7bb4b380644896204d1e5eae8b917a032179e3d13d7d87ea52c0ef00cc34fdd2R275-R285 looking particularly suspicious to me (but I haven't verified this is indeed the source of the problem)

[Cali0707]

/assign

Thanks for finding this @maschmid !!