Emit header for gRPC messages

[pcostell]

gRPC connections currently don't have enough information in the request for us to do efficient traffic management/routing.

Will you add a header that allows for us to optimize for this on the server? The header format is:

name: google-cloud-resource-prefix
value: projects/my-project-id

In general, the prefix would be the same as the JSON-over-http URL up to any custom method. For example, a Cloud Datastore commit message over JSON would hit the url projects/my-project-id:commit.

Here is an example usage in the cloud-bigtable-client: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/pull/915/files#diff-1078787e69bc1d7c5bb8507f0a87a143R83

[jskeet]

Hmm. Should be feasible, but I'll need to make sure it doesn't squash any other headers. We probably can't easily do it for the middle abstraction layer - only for DatastoreDb - but that's what we'd expect most people to use anyway. (Likewise it may be tricky to do it for users who already specify a DatastoreClient, as we basically need to override one of their settings. Will see what we can do.)

I'll probably wait until next week when @chrisdunelm is back for a spot of advice, as he knows better than I do what the best place to inject this is.

Just to check - if the value is wrong for some reason, e.g. because the user creates a DatastoreDb for one project, but then somehow issues a request for something in a different project, e.g. by looking up a specific key elsewhere, will that just be inefficient, or will it actually fail? I don't want to end up with too much logic in the client to know when to include it.

[pcostell]

We do reserve the right to fail the request if the projects don't match. Does querying for a different key actually work? I'd expect the library to populate the project ID on the request, so a lookup for a different key should fail anyways.

Could we just pull the projectId out of the request before it gets sent and tack on the header info?

[jskeet]

Does querying for a different key actually work?

I haven't tried it, but the project ID is specified separately from the keys in the API, so I'd expect it to work. (If you can only look up keys within the same project, the project ID could be inferred from the keys you're looking up...)

The problem is there isn't anywhere in the manual code to do that appropriately. We can do it in DatastoreDb using the project ID that we specify when making requests, but as I say that could end up being different from the project ID in the partition being used.

It sounds like we might need some kind of interception layer, ideally really quite low down so that it doesn't matter which abstraction layer is used. It's slightly painful that without inheritance, the code to pull the project ID out of each request needs to be done separately for each message. (We could use proto reflection, admittedly.)

Will continue mulling, and consult with Chris Bacon when he's back from holiday.

[jskeet]

@chrisdunelm for thoughts....

[chrisdunelm]

There's currently no support for per-library code specialization within the VKit layer. I'm sure this could be added, so we can inject the suitable header within VKit code, but this would take some time, and I expect would need a design that the wider vkit team are happy with.

We could potentially add a callback mechanism into vkit-gened code, allowing changes to be made to the CallSettings (which include the header) just before it's passed down into the gRPC layer.
I'm not quite clear if this would actually help in this specific case?
Although, as Jon already mentioned, this would presumably be configured on DatastoreClient (the vkit-gened client code) creation, so won't help if users are explicitly creating this client. It'll only work if users are using the DatastoreDb layer, and letting that layer create the underlying client.

[chrisdunelm]

After a quick chat with @jskeet, we've realised vkit would emit a partial method for each RPC method, which can modify the CallSettings (and possibly the request as well).
E.g. for DatastoreClient.Lookup, we'd generate a signature something like:
partial void Lookup([ref] LookupRequest request, ref CallSettings callSettings);
We then manually implement the partial method for datastore to inject the header.

This is nice in that all vkit clients will benefit.

@pcostell Do you know how other languages are approaching this?

[pcostell]

I believe all the other languages are doing this for hand-crafted libraries only at the moment, see gcloud-node: googleapis/google-cloud-node#1542.