PROTOCOL_ERROR after upgrading gRPC and Bigtable libraries

[garye]

The issue was reported here: googleapis/google-cloud-go#628

The Bigtable libraries adds a string to the metadata but otherwise doesn't do much with contexts or metadata. Any ideas what could be going on?

[dfawley]

My guess is this is likely #1219. To add the metadata, are you using metadata.FromContext in a client-side interceptor? If so, it will need to be changed to use FromOutgoingContext.

[bbassingthwaite]

I've tried explicitly to do: metadata.NewOutgoingContext(ctx, nil) with no success either. Or would you expect this not to work?

[menghanl]

Do we know what the string added to metadata by Bigtable is? Does it contain special characters (like newline)?

[dfawley]

That would clear the outgoing metadata. To add an entry to the outgoing metadata, you should do something like this:

md, _ := metadata.FromOutgoingContext(ctx)
md = metadata.Join(md, metadata.New(map[string]string{"key": "value"}))
ctx = metadata.NewOutgoingContext(ctx, md)

[garye]

@menghanl We're adding a string without special characters or newlines, and we leverage this code:
https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/bigtable/bigtable.go#L779

Looks like @dfawley has a PR to fix this for us:
https://github.com/GoogleCloudPlatform/google-cloud-go/pull/627/files

What's the status of that?

[bbassingthwaite]

So if that is the case: Why does context.Background() work but not metadata.NewOutgoingContext(ctx, nil) ?

[dfawley]

It sounds like it's something else in the context, not the metadata, then...

Can you log the context? It may also be helpful if you could provide a reproducible test case so we can dig into it.

[bbassingthwaite]

So I had tried clearing both incoming and outgoing, but never both. This line is what ended up working for me. No idea why this works, but it does:

newCtx := metadata.NewOutgoingContext(
    metadata.NewIncomingContext(ctx, nil),
    nil,
)

[dfawley]

Almost definitely what's happening here is the client interceptor that adds its metadata is reading your incoming metadata (because it is using metadata.FromContext), appending its value to the metadata there, then using that as the outgoing metadata. This is the main root cause, and that should be fixed by googleapis/google-cloud-go#627.

In addition, it seems there's characters in the incoming metadata that isn't allowed in text-only outgoing metadata, and that's why we're getting the error we're getting. It would be interesting to know the nature of the redacted authorization/authority values (e.g. are there characters in there outside the range of the base64 encoding set?). Also, you said earlier that you have clients that are running both the latest grpc version and older versions, and only the older versions fail. Can you compare the incoming metadata at the server from those two clients and see if you can discern a difference between them?

[dfawley]

The fix to the bigtable client library was merged, so this should be resolved now.