CAL data proto fields don't match actual CAL data

[grant]

Expected Behavior

I'd expect all fields that are present in in the CAL data seen for Events are present in our proto definitions. We can't have missing fields, or else our libraries will be missing fields from the JSON payload.

For example, these fields seen from Events are not found:

• @type
• authorizationInfo.resourceAttributes
• requestMetadata.callerNetwork
• requestMetadata.requestAttributes
• requestMetadata.destinationAttributes
• resourceLocation

Actual Behavior

There are many fields that are seen with CAL data, but not present here.

Here's an example of the event payload:

{
  "protoPayload": {
    "serviceName": "storage.googleapis.com",
    "methodName": "storage.buckets.update",
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "resourceName": "projects/_/buckets/test-bucket",
    "status": {},
    "authenticationInfo": {
      "principalEmail": "example@gmail.com"
    },
    "requestMetadata": {
      "callerIp": "192.168.0.1",
      "callerSuppliedUserAgent": "apitools Python/3.7.7 gsutil/4.51 (linux) analytics/disabled  google-cloud-sdk/297.0.0,gzip(gfe)",
      "callerNetwork": "//compute.googleapis.com/projects/google.com:cloudtop-prod/global/networks/__unknown__",
      "requestAttributes": {
        "time": "2020-06-18T19:05:26.688510991Z",
        "auth": {}
      },
      "destinationAttributes": {}
    },
    "authorizationInfo": [
      {
        "resource": "projects/_/buckets/test-bucket",
        "permission": "storage.buckets.update",
        "granted": true,
        "resourceAttributes": {}
      }
    ],
    "resourceLocation": {
      "currentLocations": [
        "us-central1"
      ]
    }
  }
}

Here's an example from the Knative GCP repo:

https://github.com/google/knative-gcp/blob/master/docs/examples/cloudauditlogssource/README.md

---

I'm not sure if this is because these protos are not complete after copying them over or if this is intentional.

CC @jskeet

[jskeet]

Okay, a few separate issues here:

• Why are we seeing protoPayload? I'd expect the data of the CloudEvent to just be the AuditLog message. We should talk to the event teams about that.
• The @type attribute is because this is basically a google.protobuf.Any converted into JSON. I'm not sure it's a good idea for that to be in the event at all - it's going to cause confusion - but it may be hard to remove. Again, let's talk to the event teams.
• The other "extra" fields are more easily explained: our schema hasn't yet taken this commit into account. I've been aware of this, but focusing on other event types. I'll create a PR for that today.

[grant]

Okay, a few separate issues here:

• Why are we seeing protoPayload? I'd expect the data of the CloudEvent to just be the AuditLog message. We should talk to the event teams about that.
• The @type attribute is because this is basically a google.protobuf.Any converted into JSON. I'm not sure it's a good idea for that to be in the event at all - it's going to cause confusion - but it may be hard to remove. Again, let's talk to the event teams.
• The other "extra" fields are more easily explained: our schema hasn't yet taken this commit into account. I've been aware of this, but focusing on other event types. I'll create a PR for that today.

• The protoPayload has been in the CAL since the beginning of the service and is described a bit in the private preview guides. This is what the Events for Cloud Run sees and is a core part of CAL:
  • https://cloud.google.com/logging/docs/audit/understanding-audit-logs#format_of_audit_log_entries
  • I'm not sure why we shouldn't see this. It's very clear that there is a mismatch (not a huge mismatch) with the protos and what's seen for Events.

All in all, let's be sure that the payloads seen by users exactly match the payloads fields here. I'd assume the protos would EXACTLY match the payload seen for some specific CloudEvent#type (which includes the version). This is to be used by our client libraries.

[grant]

Part of this might be an artifact of logging this information. Will need to figure out the differences with logging there.

[jskeet]

This is now being discussed internally - we'll post an update here when there's more resolution. (That's probably better than trying to maintain multiple lines of communication.)

[grant]

I'm seeing all of these fields in the proto and jsonschema, so closing. If CAL events v2 will unwrap the protoPayload field, that's a separate issue (version).