Support vNalpha in protoxform and API boosting

[htuch]

Our current workflow only supports v3 major as a target version, but folks will be wanting to merge alpha APIs ASAP (both cache and Wasm). This issue tracks adding in the requisite support for this.

[htuch]

@mattklein123 I've started to look into this, and it gets pretty complicated, pretty quickly, when we have multiple alpha and major versions coexisting. For example, do we allow v3alpha to reference another v3alpha message when a v3 message equivalent exists? What do we do when v2alpha and v3alpha get frozen to v2 and v3? Do we preserve the existing messages etc.

WDYT about moving to a model where we just have v2 and v3, and then some package-level option annotation that indicates "stuff might break, you are warned"? This simplifies back to yearly major versions, with the only alpha being the machine generated candidate for the next major version (in our case right now v4alpha).

CC @envoyproxy/api-shepherds

[mattklein123]

WDYT about moving to a model where we just have v2 and v3, and then some package-level option annotation that indicates "stuff might break, you are warned"? This simplifies back to yearly major versions, with the only alpha being the machine generated candidate for the next major version (in our case right now v4alpha).

Sure that's fine, as long as we have some way of agreeing that a proto can be changed in a breaking way. The annotation approach has the benefit of also allowing automatic documentation production along these lines.

[lizan]

I thought the plan was boosting v4alpha and add new packages in v4alpha instead of v3alpha?

What's the case for:

For example, do we allow v3alpha to reference another v3alpha message when a v3 message equivalent exists?

If there's a v3 message equivalent why there will be also a v3alpha message?

[htuch]

@lizan imagine we are adding the cache filter today. It wants to go into v2alpha, since it's still experimental. It also wants to be in v3alpha potentially. Then, at some points it gets frozen to v2 and v3.

I'm thinking what we really want is for these "alpha" protos to be part of the respective major version, just to be in a well documented "stuff might change" status.

[lizan]

@htuch what blocks it to be v3alpha, and when we boost v4alpha and v4 it could be stabilized then? I don't see any reason it need to be v2alpha especially it's an extension config. And there's still no case that "v3alpha to reference another v3alpha message when a v3 message equivalent exists"?

[htuch]

Folks want to continue to add to the v2 API today, we will allow this for Q1. They also will want their v3alpha APIs to be part of v3 when they stabilize.

[lizan]

It is totally fine to include a filter config from v3alpha in v2 Listener/HCM, no?

[htuch]

@lizan what if it references v3 core data types which also exist in v2? E.g. things in envoy.type. We are then violating ODR in some sense. This might technically work at the implementation level, but how do we represent this in documentation and explain to folks how this works? Also, what if a management server wants to start serving this proto but doesn't have any v3 base support yet?

[lizan]

@htuch Hmm, it doesn't violate ODR since at implementation level they are all v3 now even if you define a v2alpha proto at C++ level (specifically to extensions). So the only real issue is how to represent in documentation until we make all v3 docs available, right? My take is that we will have to do that by the next release so it isn't a real blocker.

Also, what if a management server wants to start serving this proto but doesn't have any v3 base support yet?

What do you mean by "v3 base support"? If a management server want to serve this proto (regardless v2 or v3), it must already have generated v3 base protos? Or do you mean a management server with filling logics of existing v2 protos?

[htuch]

I mean the management server is still using v2 envoy.type instead of envoy.type.v3.

The basic problem we're trying to solve is that we want to allow some developer to add a new filter, e.g. cache, to the APIs today, without forcing management servers to have to move to v3 types (until EOQ1).

The ODR aspect is not C++ ODR, instead it's https://github.com/envoyproxy/envoy/blob/master/api/API_VERSIONING.md#one-definition-rule-odr

[lizan]

@htuch My take of that ODR is that doesn't apply over Any boundary, I think we discussed v3 Listener/HCM contains v2 filters case before as well, and that's one of the motivation of TypedStruct so each Any can be versioned differently.

[htuch]

I think it's fine to allow v3 to be embedded in v2, but not to force the use of v3. Here's a few scenarios:

1. We allow folks to add protos to v2alpha, v3alpha, v4alpha. If we do this, they need to be able to add to the earlier version they want to support, we need protoxform to support chaining through the upgrades in an alpha world, independentish of what is going on for major version, etc. I think this is pretty complicated.
2. We only allow folks to add to v4alpha for unstable APIs. They can't freeze things until EOY, that can't work.
3. We allow folks to only add to v3alpha and v4alpha, and then to freeze to v3. This has some tooling complexity, and forces management servers to become aware of any referenced v3 types.
4. We stick to supporting only types in v2, v3, v4alpha for 2020. APIs can be added anywhere and chained forward (as with existing APIs) and we use an unstable annotation at package level to denote that they are subject to finalization.

(4) starts to sound appealing when the limitations of (1)-(3) are considered.

[lizan]

After the grace period that we still allow addition to v2 (Q2?), we won't allow either v2 nor v2alpha in either scenario right? So there will be no real difference between 1 and 3 at that time. I think we need the unstable annotation at package level anyway for either 1,3,4.

The question is whether we want to force management servers to become aware of any referenced v3 types in new extension config added during the grace period.

[htuch]

OK, I will move ahead with the annotation.

Re: awareness of v3 in the grace period, my main concern is over non-trivial protos. Things like metadata matchers etc. have a bunch of code generation behind them, so I think it could have big impact if we force this right away in order to consume new extensions. But, I think I'd be convinced either way on a case-by-case basis. It would be good to agree on a hard rule here.

[lizan]

Things like metadata matchers etc. have a bunch of code generation behind them, so I think it could have big impact if we force this right away in order to consume new extensions.

Did we make big changes between v2 and v3 for those types except packaging? I guess some sort of wirecast will work in management server as well?

[htuch]

Yes, some management servers can do this, but I think we don't want to force that yet. Even if the changes are trivial, there's a lot of boiler plate in some languages (I think Go is one of the more problematic).

[lizan]

We already started the clock deprecating v2 so I'm not keen to keep more compatibility during the grace period than we have to. For new extensions I'm inclined to have the config in v3(alpha) only and have management server do the wirecast for the new extension.

[htuch]

@lizan current advice is to add to v2/v3 at the same time in https://github.com/envoyproxy/envoy/blob/master/api/STYLE.md#adding-an-extension-configuration-to-the-api. This is changeable, I'd like to get @mattklein123's take with his "think of the management server operators" hat on.

[mattklein123]

For brand new extensions it probably doesn't matter, so I would just go with whatever is easiest. I don't have an issue with v2/v3 as long as we have the annotation in place that says we can change the protos and they are unstable.

[lizan]

The issue of v2/v3 without alpha package is you can't tell whether that's a stabilized config from Any or TypedStruct, which makes hard to debug config.