deprecate config using the One True ADS Server(tm)

[markdroth]

Today, the way ADS is configured is as follows:

• The bootstrap file has an ads_config field indicating the One True ADS Server(tm).
• The bootstrap file has its lds_resource_locator and cds_resources_locator fields set to a ConfigSource with the ads field set.
• In resources sent from the management server, ConfigSource protos use the same ads field.

This approach seems incompatible with the new xDS naming scheme, which was designed to support federation. As part of the new naming scheme, the ApiConfigSource.ApiType enum now has AGGREGATED_GRPC and AGGREGATED_DELTA_GRPC values, which allows configuring any ConfigSource to use a single ADS stream instead of using a separate stream for each resource type. This way of configuring ADS seems much more in-line with the direction we're heading in and will make it easier to transition to the new federated world.

Therefore, I propose replacing this notion of the One True ADS Server(tm) with use of this new AGGREGATED_GRPC enum value. The typical ADS configuration described above would instead be represented as follows:

• The bootstrap file will not populate the ads_config field.
• The bootstrap file will have its lds_resource_locator and cds_resources_locator fields both set to the same server, with ApiType set to AGGREGATED_GRPC or AGGREGATED_DELTA_GRPC. (Envoy will internally de-dup these by hashing the ConfigSources, so the result will be a single ADS stream, not two separate ADS streams.)
• In resources sent from the management server, ConfigSource protos will use the self field.

This will require the following changes:

• Implement support for the self field. (This field was added in xds: Add self config source type. #8201 but is not currently implemented in Envoy.)
• Implement support for the AGGREGATED_GRPC and AGGREGATED_DELTA_GRPC values in the ApiType enum. (I'm not sure how much of this works right now.)
• Deprecate the bootstrap file's ads_config field.
• Deprecate the ads field.

The only configuration that would be broken by this change would be if someone was (e.g.) using non-aggregated LDS to get a Listener resource that contained a ConfigSource telling the client to get the RouteConfiguration via the One True ADS Server(tm). However, it seems exceedingly unlikely that anyone is actually doing that, and there are work-arounds: they can either have their management server send the exact ConfigSource containing the ADS server they want to talk to, or better yet, they can start using the new xDS naming scheme, where they can select the xDS server via an authority in the resource name itself.

CC @mattklein123 @htuch

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[htuch]

I agree with this directionally, suggest waiting until we have minor versioning working to execute.

[nezdolik]

@markdroth @htuch is anyone working on this? i can help out

[markdroth]

I think @adisuissa was going to look at this, but I don't know if he's started yet.

[adisuissa]

In addition to the problem with non-aggregated listeners described in the first comment, I think there are other use-cases that may not fit with the proposal. Specifically, self limits which ADS config to use. Here are a couple of examples:

1. Using static clusters, where different sub-groups of clusters each wants to use a different ADS config source for their endpoints.
2. Using one ADS for clusters, endpoints, listeners, etc. and another ADS for secrets.

A proposal for named config-source that gets around this issue, and suggests a path forward on future xDSTP features, can be found here.

[markdroth]

@adisuissa I think both of those use-cases are actually already handled by the approach described in this issue. I don't see how the proposal for named ConfigSources actually helps with any of this.

Note that there is no requirement here to use "self" ConfigSources for ADS. The "self" option is useful in that it prevents the need for control planes to embed their own name in their resources, but that option really has nothing to do with ADS specifically. The intention here is that a ConfigSource can specify any server it wants and can request the use of ADS by setting the ApiType enum to AGGREGATED_GRPC, so it should be possible to get whatever resources you want from whatever ADS streams you want.

[adisuissa]

Note that there is no requirement here to use "self" ConfigSources for ADS. The "self" option is useful in that it prevents the need for control planes to embed their own name in their resources, but that option really has nothing to do with ADS specifically. The intention here is that a ConfigSource can specify any server it wants and can request the use of ADS by setting the ApiType enum to AGGREGATED_GRPC, so it should be possible to get whatever resources you want from whatever ADS streams you want.

My concern is the need to rewrite the same config source in many places, and then the need to identify the same source by comparing the protos (it can be hashed, but not ideal). Making the names explicit has the benefit of defining it once and referencing it from multiple places.

[markdroth]

If the goal is to avoid the repetition, then using xdstp names gives you that, where the name is the authority. Why do we need a second mechanism to do the same thing?

[adisuissa]

I agree that xdstp naming gives that, but requires changes on the server side.
In addition the named-config-source allows decoupling the stream from the authority.
For example, if a single authority provides both secrets and other config, but wants to serve them from different servers, then the authority in the resource names can stay the same, but the providing server could be different.

[markdroth]

The named ConfigSource approach also requires server-side changes, so I don't see how that's a benefit here.

The whole point of the authority in the xdstp names is to indicate what server to get the resource from. There's no value in having the same authority for two resources that you want to fetch from different servers.

I don't see what this approach gives us that we don't already have. I think we should implement the thing we've already designed rather than add a second way to do the same thing.

[adisuissa]

I was under the impression that the server side changes for the new xdstp-naming scheme is non-trivial, but I may be wrong.
The named-config-sources requires modifying just the ConfigSource (which is also required by the approach described in the first comment), while also allows reusing the ADS stream in non-xdstp naming use-cases.

[htuch]

xdstp:// changes aren't really the huge IMHO. You can even start by using them as opaque identifiers in server or client.