Moving to cev2 - publisher, receive adapter - Implementing KRShaped

[nachocano]

Fixes #1167
Fixes #1166
Helps with #1209

Proposed Changes

• 🧽 Moving publisher and receive adapter to cev2 and using wire.

• 🧽 Updating some of the CE formats due to ongoing conversations. There were undocumented extensions we were adding that are not needed. The release notes of @yolocs PR should include all the breaking changes we are planning to make.

• 🧽 Implementing KRShaped in all resources otherwise it does not compile

• TODOs

  • Fix adapter UTs to be done in a follow up as it might take sometime and don't want to block @capri-xiyue nor @yolocs Add UTs for newer Receive Adapter #1333

• Move e2e tests to cev2. E2E tests should use CE-SDK v2 #1336

Release Note

We are using CE SDK v2 for Sources and Channels, with all its performance improvements

Docs

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nachocano

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nachocano]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nachocano]

/assign @Harwayne
/assign @ian-mi
/assign @yolocs

Guys this is a WIP PR, but it is kind of big so I want to get your input asap. Especially on some ugly part of our old receive adapter that IIRC we use for channels. There is a tracing part that I'm not sure what we are actually doing...

@yolocs this will help out with the changes to events schemas. The strings hasn't been changed yet, but I refactored things so that it would be easy to do...

Either way, I'm having seconds thoughts about including this as it's big and kind of last minute. We can always do the changes of the events without moving to cev2, and then do the move...
The upside of doing this now is that I've heard from folks that the old sdk was not very performant. So this might improve the performance of our sources data plane, which we have been neglecting so far.

[nachocano]

/retest

…

On Wed, Jun 24, 2020 at 12:43 AM Knative test reporter robot < ***@***.***> wrote: The following jobs failed: Test name Triggers Retries pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests <https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/google_knative-gcp/1296/pull-google-knative-gcp-wi-tests/1275660442689081345> pull-google-knative-gcp-wi-tests <https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/google_knative-gcp/1296/pull-google-knative-gcp-wi-tests/1275668746555363328> pull-google-knative-gcp-wi-tests <https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/google_knative-gcp/1296/pull-google-knative-gcp-wi-tests/1275677807183138816> pull-google-knative-gcp-wi-tests <https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/google_knative-gcp/1296/pull-google-knative-gcp-wi-tests/1275687369739079685> 3/3 Job pull-google-knative-gcp-wi-tests expended all 3 retries without success. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1296 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABD65DCI7LHFD3EUXO5YTPTRYGVDTANCNFSM4OAXLQIQ> .

[nachocano]

/test pull-google-knative-gcp-wi-tests

[nachocano]

/test pull-google-knative-gcp-wi-tests

[nachocano]

pull-google-knative-gcp-wi-tests is failing pretty much all the time. I'm thinking of disabling them if folks are having the same issues in other PRs

[nachocano]

Seems WI tests are not required to merge...
Waiting until they run one more time, and the UTs as well (there was a flake on the configMap assertion).
I think we can get this in to unblock people and I can focus on the follow ups... @yolocs @ian-mi

[nachocano]

/test pull-google-knative-gcp-wi-tests

[knative-test-reporter-robot]

The following jobs failed:

Test name	Triggers	Retries
pull-google-knative-gcp-wi-tests	pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests	3/3

Job pull-google-knative-gcp-wi-tests expended all 3 retries without success.

[ian-mi]

I would have preferred the KRShaped change to be done in a separate PR since these are essentially two large orthogonal changes, but I am ok with merging these together in order to unblock things.

[yolocs]

/lgtm

[nachocano]

/test pull-google-knative-gcp-unit-tests

[nachocano]

Agreed with @ian-mi and @yolocs to get this in and I'll address the unresolved comments in the follow up.
We need to unblock @yolocs work and allow @capri-xiyue to get new numbers on receive adapter resource requests and limits

[nachocano]

@ian-mi addressed the tests comments so that I don't forget. @yolocs will have to do changes either way and I'll also improve stuff with the follow up

[nachocano]

Need one more lgtm

[yolocs]

/lgtm

[ian-mi]

/lgtm
Thanks for the changes!

[knative-metrics-robot]

The following is the coverage report on the affected files.
Say /test pull-google-knative-gcp-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/events/v1beta1/cloudauditlogssource_types.go	75.0%	80.0%	5.0
pkg/apis/events/v1beta1/cloudbuildsource_types.go	71.4%	77.8%	6.3
pkg/apis/events/v1beta1/cloudpubsubsource_types.go	86.7%	88.2%	1.6
pkg/apis/events/v1beta1/cloudschedulersource_types.go	71.4%	77.8%	6.3
pkg/apis/events/v1beta1/cloudstoragesource_types.go	71.4%	77.8%	6.3
pkg/apis/intevents/v1beta1/pullsubscription_types.go	92.9%	93.8%	0.9
pkg/broker/handler/providers.go	60.0%	75.0%	15.0
pkg/pubsub/adapter/adapter.go	36.5%	0.0%	-36.5
pkg/pubsub/adapter/context/project.go	Do not exist	100.0%
pkg/pubsub/adapter/context/subscription.go	Do not exist	100.0%
pkg/pubsub/adapter/context/topic.go	Do not exist	100.0%
pkg/pubsub/adapter/converters/auditlogs.go	75.0%	75.6%	0.6
pkg/pubsub/adapter/converters/build.go	100.0%	88.2%	-11.8
pkg/pubsub/adapter/converters/converters.go	91.7%	71.4%	-20.2
pkg/pubsub/adapter/converters/pubsub.go	96.0%	83.3%	-12.7
pkg/pubsub/adapter/converters/pubsub_pull.go	Do not exist	84.2%
pkg/pubsub/adapter/converters/scheduler.go	92.9%	93.8%	0.9
pkg/pubsub/adapter/converters/storage.go	87.5%	94.4%	6.9
pkg/pubsub/adapter/providers.go	Do not exist	0.0%
pkg/pubsub/adapter/stats_reporter.go	81.8%	83.3%	1.5
pkg/reconciler/events/storage/storage.go	72.5%	72.2%	-0.3
pkg/reconciler/intevents/controller.go	50.0%	100.0%	50.0
pkg/reconciler/intevents/pullsubscription/resources/receive_adapter.go	87.1%	88.9%	1.8
pkg/reconciler/intevents/reconciler.go	78.4%	78.8%	0.4

[knative-prow-robot]

@nachocano: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-google-knative-gcp-go-coverage	014802a	link	/test pull-google-knative-gcp-go-coverage

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.