Attendees:
- Kevin Gibbons (KG)
- Peter Hoddie (PHE)
- Shu-yu Guo (SYG)
- Richard Gibson (RGN)
KG: Should we pursue this at all?
PHE: I put together some notes. I think in the broader sense this is worth spending time on. My concern is I don't think TC39 is the right venue for that. After the meeting I looked at how base64 is handled today, and you're absolutely right to be horrified by that and want to do better. I looked around and there's no shortage of libraries that do base64. There's not a question there's some need here. The web's approach is by far the worst of them all IMO since it happened before ArrayBuffers. Interop would be welcome. That's all common ground. Question is whether it should be in TC39. One of my favorite obscure documents is the Extensible Web Manifesto. I think that gives good guidance on how priority should be set in a busy world. First question I ask: base64, do you need the engine, do you need the browser to do it well? My conclusion was no, based on practical experience that we have libraries like Pako (?) that does zlib compression that has great throughput. I don't think raw performance of a native implementation is necessary. IIRC you were talking about a 1-shot call so you weren't looking at volume of data. So what we need is a common API and that would benefit ecosystem. That common API is actually kind of complicated when we get to streaming. I looked at do we need streaming, and from Moddable's perspective our API is 1-shot right now and it's fine. But there are cases where our base64 is insufficient; if there's an interoperable streaming API we can move away from our 1-shot. Another thing that came to mind that base64 encoding is similar to text encoding and decoding. Not identical, but not unrelated. And we don't have text encoding and decoding in the language, and we would argue it's more useful. Why is that? Reason is web got there first with TextEncoder/Decoder. Node's been adopting it, and Moddable has an implementation. It works well and it's nicely designed.
KG: I dispute that.
PHE: Okay well I don't love it, but it's more than adequate. It's adequate in a way the base64 support in the browser is certainly not. I'd be pleased and enthusiastic to spend time on the API in a forum where the API would be adopted by the JS ecosystem. But for the reasons I've gone through I don't think it's consistent with prioritization that it lands in the language. It'd be a subset of the functionality we need anyway for streaming in a constrained environment. It wouldn't solve the one base64 problem we have today.
KG: Let me summarize. You agree base64 is broadly useful, currently inadequately supported in the platform, and you'd like to see it available across JS ecosystems, but you don't think TC39 is the right venue to pursue that. That's the part I'm mosing missing, you don't think TC39 is the right venue because… extensible web manifesto arguing for lower level first?
PHE: I think to do base64 properly we covered a lot of subtle points about base64 and they're tedious but necessary. And there's the streaming support which I do think is essential. Once we get all that in place it's a big thing, and a bigger thing than TC39 needs to spend time on. We didn't bite off text encoding. If you were taking a pure look at what's missing in the language, that'd be a more appropriate place. But that evolved to an available API without being part of the language.
KG: I have 2 major responses. First, for me, things that should be across JS ecosystem and things that things that TC39 should deal with are synonymous. It's strange to me to say a thing should be available to JS ecosystem is exactly TC39's mandate. I see things like WHATWG stuff like text encoding to be a failure of TC39 process where due to lack of functionality WHATWG took it into their own hands. Then everyone else copies it, at least Node, because once it's in WHATWG it's de facto standard. But definitely seems like a failure of TC39.
SYG: Organizationally TC39 hasn't successfully scaled up for speccing richer stdlib stuff. For smaller stuff this is fine, for more complex stuff we run into prioritization concerns PHE raised.
KG: If we don't do this in TC39 one of three outcomes happens: this doesn't become available to JS programmers, this becomes available via WHATWG, whose design sense is atrocious, and (??? missed 3rd option)
KG: <the ways in which WHATWG-designed Base64 decoder would be different worse, sorry I missed it>
PHE: To your point, I don't do web development so WHATWG's bad designs… no lost love there. I don't think there're two choices here. There are plenty of JS APIs that evolved outside of any standard organizations used. I think there's an opportunity here: here's something that crosses boundaries and need proper API design. I don't think it's quite as black and white as if TC39 doesn't do it WHATWG will and that will be unpleasant.
KG: It is very definitely true if TC39 doesn't do it WHATWG will pick it up. There's already a WHATWG proposal. If I go back and say TC39 doesn't do it, WHATWG will do it and Chrome will just ship it without much input. If TC39 is unwilling to do it, I guarantee WHATWG will do it and I will even push for it as a web developer.
There isn't a venue for doing API design that should be available across ecosystems other than TC39. Whether we do it in TC39 or in WHATWG and everyone copies it.
PHE: Not sure what the option here is. One way to look at it there's some race or venue debate. It's kind of silly, like, they all want the same thing. TC39 isn't going to decide it's going to do this.
SYG: Can we charter something short-lived to design this out-of-band with plenary, in deference to prioritization concerns.
KG: I don't think committee will go for that.
PHE: Is one of your concerns that WHATWG's process and design sense is unpleasant?
KG: Not that it's unpleasant but marginally worse. I want static methods on ArrayBuffer, and that's not a thing that WHATWG does and is precisely a thing TC39 does. If there's a Base64Encoder global, that's not the end of the world but the marginally worse. Point two is WHATWG only ships things in secure concerns and that is really annoying.
PHE: I think it expands what the scope of TC39 is. This seems to be in what's called in the old world standardization "harmonization". Maybe there's more work to do here to come up with something that meets the requirements that if WHATWG just adopted it would be bad.
KG: With my web developer hat on, the main difference between TC39 and WHATWG is where the API lives. If it's in TC39 it can be on ArrayBuffer and I want it there. If I don't care about where it is I don't care where the venue is. I think we should decide whether it's a thing that belongs in the language or not. If it does belong, we should pursue it the same way we pursue everything else.
PHE: What's your read of WHATWG?
KG: I think if they were to wait 2 years they'd just ship something, justifiably. I'm interested in a go/no go in TC39 decision.
PHE and KG:
PHE: As a point of reference from our world, the Stream that exists in WHATWG that don't exist in TC39 are problematic. No objection to streams. It seems exceedingly well designed but it's a little fancy for our world. Bizarreness of TextEncoder's way dealing with splits and streams is great in that it doesn't import Streams. I presume you're not proposing to bring Streams to TC39. In that respect we share an API goal.
KG: I don't know if they'd copy TextEncoder outright or do a DOM stream or do both. If it were up to me I'd want the actual Stream one so I can do it on file uploads. I hope they'd make the streams available.
PHE: For streams we'd like to see something more factored.
KG: Well do you want to pursue this in TC39 then?
PHE: You're at Stage 1, is that right? Stage 1 says we're going to study this further. I didn't object to Stage 1 and happy to provide input to that. I think we made progress in understanding each other which is great. I don't know where it goes. I don't know how to sort things out with WHATWG. A good outcome would seem to be we have one solution here if at all possible.
KG: With my web dev hat on I really do want two things. I want a sync API. And I want a transform stream for streaming things on the web platform.
PHE: But you don't want two totally unrelated things to do that?
KG: I kinda would! They're just very different shaped things. A sync API is a function that I call somewhere, put data in get data out. A transform stream is like an object I instantiate and pipeline data through using streams API. Neither is particularly well suited for doing the thing the other one is doing.
PHE: But someone would write the glue once and you'd use that?
KG: I would not use it I'd just write the 5 lines.
PHE: If it's 5 lines they can get it from search anyways it's not a big problem.
KG: Well, I guess writing a transform stream correctly is genuinely tricky due to backpressure, so probably more than 5 lines.
PHE: So you'd use somebody else's?
KG: I'm reluctant to say it's fine people will implement it themselves. My experience it gets implemented well enough to work for 90% but it's not correct. But it breaks in terrible ways in last 10% and we just live with that forever.
PHE: But then WHATWG can come in and do the transform stream. That would be fine.
KG: Yes, to be clear that's my preferred outcome. TC39 does sync API, WHATWG does stream API. They feel like they don't belong in the same API because they feel pretty different. For my use this is fine, I understand you want stream solved in a way that doesn't involve transform streams.
I think you might be right that if we say this API needs to solve for streaming it'll grow too large to be effectively solved in TC39. If we say that WHATWG will do a transform stream, and no one else will pursue a streaming API because Node will get that too and that'll be the end of it.
PHE: I don't have the experience or visibility. Maybe there's a finesse way to get a simple API but I'm not sure.
KG: I'm hopeful we can come up with an API… multi-shot, let's say, that'll meet Peter's needs that won't be unreasonable to do in TC39.
KG: Other requirements? Async is a big question, streams is a medium question. Other ones are small where we just make a choice.
PHE: Other concerns like whitespace things and weird alphabet things like you mentioned. Any text parsing thing has to deal with 50 years of history and it can be surprisingly tricky.
KG: So if we work out streaming are you comfortable with doing it in TC39?
PHE: I'm still not wild about it.
KG: If the outcome is you don't think it's useful for the language that's fine. I prefer to know relatively soon so I can work with WHATWG instead.
PHE: Let me put it this way. If there's a lot of support in committee and the API design is good I'm not going to stand in the way. If it's lukewarm we should think twice. I'll freely admit my sensibility of the language are very different from web developers'.
Conclusion: Exploration of a multi-shot base64 API that meets Peter's use case needs as well as for Peter feeling comfortable to be in scope for TC39.