Update sources event CE types, sources, subjects #1340

yolocs · 2020-06-24T21:48:54Z

Proposed Changes

Used new values for CE type/source/subject
Scheduler event data schema is updated
Unbundled event schemas from API versions

Release Note

BREAKING CHANGES

- All event types are updated to match event types in googleapis/google-cloudevents
- CAL/Storage/Scheduler events now also have updated CE "source" and "subject"
- Scheduler events now also have a new data schema

ACTION REQUIRED

- Users who only depend on "source", "type" and "subject" values in the trigger filters can create new triggers with new filter values (see details below).
- Users who depend on "source", "type" and "subject" values in their receiver code will have to update their code.
- Users who depend on scheduler event data will have to update their code.

DETAILED CHANGES

1. All event types and data schemas were updated to be compliant with https://github.com/googleapis/google-cloudevents/tree/master/proto/google/events/cloud. There is no real change to the CAL/PubSub/Storage event schema. Scheduler event schema was updated to be https://github.com/googleapis/google-cloudevents/blob/master/proto/google/events/cloud/scheduler/v1/data.proto
2. Source changes:
  CAL: was "//[service_name]/projects/[project_id]" => now "//cloudaudit.googleapis.com/projects/[project_id]/logs/[activity/data_access]"
  Scheduler: was "//cloudscheduler.googleapis.com/projects/[project_id]/locations/[location]/schedulers/[k8s_scheduler_source_name]" => now "//cloudscheduler.googleapis.com/projects/[project_id]/locations/[location]/jobs/[job_name]"
  Storage: was "//storage.googleapis.com/buckets/[bucket_name]" => now "//storage.googleapis.com/projects/_/buckets/[bucket_name]"
3. Subject changes:
  CAL: was "[resource_name]" => now "[service_name]/[resource_name]"
  Scheduler: was "jobs/[job_name]" => now NONE
  Storage: was "[object_id]" => now "objects/[object_id]"
4. Type changes:
  CAL: was "com.google.cloud.auditlog.event" => now "google.cloud.audit.log.v1.written"
  PubSub: was "com.google.cloud.pubsub.topic.publish" => now "google.cloud.pubsub.topic.v1.messagePublished"
  Scheduler: was "com.google.cloud.scheduler.job.execute" => now "google.cloud.scheduler.job.v1.executed"
  Storage:
    was "com.google.cloud.storage.object.finalize" => now "google.cloud.storage.object.v1.finalized"
    was "com.google.cloud.storage.object.delete" => now "google.cloud.storage.object.v1.deleted"
    was "com.google.cloud.storage.object.archive" => now "google.cloud.storage.object.v1.archived"
    was "com.google.cloud.storage.object.metadataUpdate" => now "google.cloud.storage.object.v1.metadataUpdated"

Docs

knative-prow-robot · 2020-06-24T21:48:56Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

knative-prow-robot · 2020-06-24T21:48:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: yolocs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [yolocs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

yolocs · 2020-06-24T22:17:48Z

/test all

nachocano · 2020-06-24T22:26:13Z

pkg/apis/events/v1beta1/cloudauditlogssource_types.go

+	CloudAuditLogsSourceLogWrittenEventType = "google.cloud.audit.log.v1.written"
+	CloudAuditLogsSourceEventDataSchema     = "https://github.com/googleapis/google-cloudevents/blob/master/proto/google/events/cloud/audit/v1/events.proto"


can we put all of these constants in register.go maybe? outside v1beta1... When we do v1, we will be copy/pasting them otherwise. Similar thing happened with the current values... They are both in v1alpha1 and v1beta1

That's a good point!

I actually moved this out of the apis folder. I feel it makes more sense as they eventually should be replaced by the code generated from the protos.

pkg/apis/events/v1beta1/cloudpubsubsource_types.go

pkg/pubsub/adapter/converters/scheduler.go

yolocs · 2020-06-25T18:55:49Z

Still trying to make sure the e2e tests are not broken

yolocs · 2020-06-25T19:02:23Z

/test all

yolocs · 2020-06-25T19:11:33Z

/retest

yolocs · 2020-06-25T19:23:03Z

/test all

yolocs · 2020-06-25T19:29:19Z

/test all

yolocs · 2020-06-25T21:32:06Z

/test all

yolocs · 2020-06-25T22:01:44Z

/test pull-google-knative-gcp-integration-tests

yolocs · 2020-06-25T22:24:14Z

/test all

nachocano · 2020-06-26T05:56:44Z

Is this still in draft? Can we get this in and maybe do follow ups if needed? @zargarpur would benefit from some of this constants in a change he is doing for metrics...
I made a pass and I liked things moved to pkg/schemas

knative-metrics-robot · 2020-06-26T16:46:34Z

The following is the coverage report on the affected files.
Say /test pull-google-knative-gcp-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/events/v1alpha1/cloudauditlogssource_types.go	75.0%	66.7%	-8.3
pkg/apis/events/v1alpha1/cloudpubsubsource_types.go	86.7%	85.7%	-1.0
pkg/apis/events/v1alpha1/cloudschedulersource_types.go	71.4%	66.7%	-4.8
pkg/apis/events/v1alpha1/cloudstoragesource_types.go	71.4%	66.7%	-4.8
pkg/apis/events/v1beta1/cloudauditlogssource_types.go	80.0%	75.0%	-5.0
pkg/apis/events/v1beta1/cloudpubsubsource_types.go	88.2%	87.5%	-0.7
pkg/apis/events/v1beta1/cloudschedulersource_types.go	77.8%	75.0%	-2.8
pkg/apis/events/v1beta1/cloudstoragesource_types.go	77.8%	75.0%	-2.8
pkg/pubsub/adapter/converters/auditlogs.go	75.6%	76.1%	0.5
pkg/pubsub/adapter/converters/scheduler.go	93.8%	91.7%	-2.1
pkg/schemas/v1/auditlogs.go	Do not exist	100.0%
pkg/schemas/v1/pubsub.go	Do not exist	100.0%
pkg/schemas/v1/scheduler.go	Do not exist	100.0%
pkg/schemas/v1/storage.go	Do not exist	100.0%

nachocano · 2020-06-26T17:11:15Z

/lgtm

I reviewed yesterday. Just looked at the updates, looks great! Thanks a lot @yolocs for doing this.

Can we create a new issue to update the examples in this repo? and we can probably ask some new member to help with that?

Similarly, I think we need to discuss what are we going to do with current OSS users... Whether we are planning on updating their Triggers (if any is filtering based on old info), or we just have a huge Action Required in the notes...
I think for now I'd add an Action Required, breaking change in the release notes, and we can discuss with the group.

yolocs · 2020-06-26T17:26:10Z

#1355 for doc updates.

grantr · 2020-06-26T17:30:53Z

Thanks @yolocs!The summary of the string changes in the release notes is great. I wrote this summary of the type strings changes to add as well:

Type changes:
CAL: was "com.google.cloud.auditlog.event" => now "google.cloud.audit.log.v1.written"
PubSub: was "com.google.cloud.pubsub.topic.publish" => now "google.cloud.pubsub.topic.v1.messagePublished"
Scheduler: was "com.google.cloud.scheduler.job.execute" => now "google.cloud.scheduler.job.v1.executed"
Storage:
  was "com.google.cloud.storage.object.finalize" => now "google.cloud.storage.object.v1.finalized"
  was "com.google.cloud.storage.object.delete" => now "google.cloud.storage.object.v1.deleted"
  was "com.google.cloud.storage.object.archive" => now "google.cloud.storage.object.v1.archived"
  was "com.google.cloud.storage.object.metadataUpdate" => now "google.cloud.storage.object.v1.metadataUpdated"

yolocs · 2020-06-26T18:57:56Z

/retest

yolocs · 2020-06-26T19:39:54Z

/retest

yolocs · 2020-06-26T22:17:24Z

/retest

knative-test-reporter-robot · 2020-06-26T22:17:34Z

The following jobs failed:

Test name	Triggers	Retries
pull-google-knative-gcp-integration-tests	pull-google-knative-gcp-integration-tests pull-google-knative-gcp-integration-tests pull-google-knative-gcp-integration-tests pull-google-knative-gcp-integration-tests pull-google-knative-gcp-integration-tests	3/3
pull-google-knative-gcp-wi-tests	pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests pull-google-knative-gcp-wi-tests	3/3

Job pull-google-knative-gcp-wi-tests expended all 3 retries without success.

Update sources event CE types, sources, subjects

373d1a9

yolocs requested a review from nachocano June 24, 2020 21:48

knative-prow-robot added the do-not-merge/work-in-progress label Jun 24, 2020

knative-prow-robot requested a review from liu-cong June 24, 2020 21:48

knative-prow-robot added approved size/L labels Jun 24, 2020

nachocano reviewed Jun 24, 2020

View reviewed changes

pkg/apis/events/v1beta1/cloudpubsubsource_types.go Outdated Show resolved Hide resolved

nachocano reviewed Jun 24, 2020

View reviewed changes

pkg/pubsub/adapter/converters/scheduler.go Show resolved Hide resolved

separate schemas from eventing api

80de674

googlebot added the cla: yes (override cla status due to multiple authors bug) label Jun 25, 2020

knative-prow-robot added size/XXL and removed size/L labels Jun 25, 2020

yolocs added 2 commits June 25, 2020 19:22

fix v1alpha1 api

be46bb8

Merge branch 'master' into f/newtypes

7368f25

fix test

9c54288

fix yamls

0325bee

fix test images for cal and scheduler targets

73e02fb

nachocano mentioned this pull request Jun 25, 2020

Limit the values of event_type in Broker metrics #1352

Merged

yolocs added 2 commits June 26, 2020 16:41

cleanups

496b06c

Merge branch 'master' into f/newtypes

3619fb1

yolocs marked this pull request as ready for review June 26, 2020 16:41

knative-prow-robot removed the do-not-merge/work-in-progress label Jun 26, 2020

yolocs mentioned this pull request Jun 26, 2020

Update docs for new CE types/sources/subjects #1355

Closed

knative-prow-robot assigned nachocano Jun 26, 2020

knative-prow-robot added the lgtm label Jun 26, 2020

knative-prow-robot merged commit 11a70b6 into google:master Jun 26, 2020

grantr mentioned this pull request Jun 29, 2020

Update to new GCP CloudEvent types for 0.16 meteatamel/knative-tutorial#73

Closed

nachocano mentioned this pull request Aug 20, 2020

Change EventType of CloudBuildSource #1608

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update sources event CE types, sources, subjects #1340

Update sources event CE types, sources, subjects #1340

yolocs commented Jun 24, 2020 •

edited

Loading

knative-prow-robot commented Jun 24, 2020

knative-prow-robot commented Jun 24, 2020

yolocs commented Jun 24, 2020

nachocano Jun 24, 2020

yolocs Jun 24, 2020

yolocs Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

nachocano commented Jun 26, 2020

knative-metrics-robot commented Jun 26, 2020

nachocano commented Jun 26, 2020

yolocs commented Jun 26, 2020

grantr commented Jun 26, 2020

yolocs commented Jun 26, 2020

yolocs commented Jun 26, 2020

yolocs commented Jun 26, 2020

knative-test-reporter-robot commented Jun 26, 2020

		CloudAuditLogsSourceLogWrittenEventType = "google.cloud.audit.log.v1.written"
		CloudAuditLogsSourceEventDataSchema = "https://github.com/googleapis/google-cloudevents/blob/master/proto/google/events/cloud/audit/v1/events.proto"

Update sources event CE types, sources, subjects #1340

Update sources event CE types, sources, subjects #1340

Conversation

yolocs commented Jun 24, 2020 • edited Loading

Proposed Changes

knative-prow-robot commented Jun 24, 2020

knative-prow-robot commented Jun 24, 2020

yolocs commented Jun 24, 2020

nachocano Jun 24, 2020

Choose a reason for hiding this comment

yolocs Jun 24, 2020

Choose a reason for hiding this comment

yolocs Jun 25, 2020

Choose a reason for hiding this comment

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

yolocs commented Jun 25, 2020

nachocano commented Jun 26, 2020

knative-metrics-robot commented Jun 26, 2020

nachocano commented Jun 26, 2020

yolocs commented Jun 26, 2020

grantr commented Jun 26, 2020

yolocs commented Jun 26, 2020

yolocs commented Jun 26, 2020

yolocs commented Jun 26, 2020

knative-test-reporter-robot commented Jun 26, 2020

yolocs commented Jun 24, 2020 •

edited

Loading