encoding/gob, encoding/json: add streaming interface

[rgooch]

When encoding a large data structure (such as a tree representing a file-system), the Encode() methods first marshal the encoded data into a []byte buffer and then write the buffer to the writer. Similarly, the Decode() methods require a buffer to unmarshal from. These buffers must be allocated (and grown) and typically more than double the memory consumption of the process. This is limiting for applications which are sized to the machine/container.

I propose that the interfaces are extended to support streaming encoding/decoding of data. For each type (except slices and maps), the data would be separately marshaled and written. For slices and maps, each entry would be marshaled and written, rather the slice/map as a whole.

Some API suggestions/alternatives:
func Encode(w io.Writer, v interface{}) error
func NewStreamingEncoder(w io.Writer) *Encoder
func (enc *Encoder) StreamingEncode(v interface{}) error

[robpike]

While I understand the desire, one of the strengths of gobs is that the receiver always knows how much buffer to allocate because the message has a header that says how long it is. That can't be done without storing the whole message in memory in the encoder.

[rgooch]

Yes, I see how that's been helpful for the net/rpc package: knowing the size means that you can multiplex RPCs over a single connection. But there are plenty of situations in which the receiver doesn't care how big the message will be. It's even more the case with the current version of the language since there is no reliable way to know if an allocation will fail, so a receiver can't make an informed decision on whether to accept or drop that 1 GiB incoming transmission.

I'm not suggesting changing the current properties. I'm suggesting an extension that makes the package useful for large data structures. It could just as well be new packages like "sgob" and "sjson" but that would result in a lot of duplicate code.

[tejorupan]

#6569 net/rpc: Streaming API for RPC

[ianlancetaylor]

Related to #11046 for encoding/json.

[mvdan]

#33714 goes into a lot more detail (and even an implemented design) for json. For the sake of keeping discussion in one place, I'm going to close this issue in favor of that one.

I realise this issue is also about encoding/gob, though it does not include a specific proposal and it hasn't gotten any traction. I think someone should open an issue specific to encoding/gob if they still strongly feel that it should have streaming support.