Proposal: Implement message keys as bytes instead of strings

[burdiyan]

This is rather a proposal for future versions, as it will be a breaking change.

The thing is that, message keys in Kafka are bytes, so can be not only simple strings, but also complex structures encoded as bytes.

For example we use Debezium (http://debezium.io) for doing Change Data Capture from MySQL databases, and Debezium uses table's primary key as message key for Kafka. It is commonly encoded with Avro as a struct like {"id": 1}.

Right now in order to deal with this, we need to pass goka.UpdateCallback everywhere, that would convert key string back to []byte, deserialize it to struct and then store only the int value converted to string as a table's key.

Using []byte for keys would just be more clear semantically for many Kafka users, since that is how Kafka works internally.

[frairon]

Initially we implemented storing keys as bytes as well, but it turned out that we never had any use case for bytes and tried to avoid the continuous casting of strings for convenience.

I could imagine implementing a key conversion based on codecs. That way each topic could have individual ways of storing and accessing their keys. All functions would then be be passed just an interface{} like with the values.
But as you said, it's a breaking change (or add many similar-looking functions to the interface), and probably not too urgent at the moment.

Many thanks for giving that hint anyways!

[burdiyan]

I actually ended up using strings all the time (it turnes out there is a way to configure Kafka Connect inputs to extract the field from the key structure). Basically I also think it's rare to use complex structures for message keys. So I'd rather close this issue.