Simplify Envoy Config Generation

[RyanTheOptimist]

The system that Envoy Mobile uses to generate Envoy's config is fairly complicated. It involves snippets of YAML defined as C strings in library/common/config/config.cc. In order to avoid duplication, these snippets make use of YAML anchors and aliases. These YAML snippets are conditionally stitched together along with in Swift via EnvoyConfiguration.m and in Java via EnvoyConfiguration.java and in C++ via library/cc/engine_builder.h.

It seems undesirable to have 3 different builders with different functionality. The Alternate Protocols cache filter is only supported in Java. Custom filters are not supported in the C++ builder.

In addition, the use of YAML anchor and aliases interacts a bit poorly with the builders. Some of the anchors are used by aliases in config_header which prevents them from being overridden by builder-supplied values. Of course, other anchor are overrideable and some are currently overrideable in some places but not in others.

I think we should simplify this. My strawman would be that config generation is implemented in a single place in (C or C++). The Java, Swift and C++ Builder would be responsible for calling into this single config generation method with whatever state is required to generate a valid config. (This includes numerous booleans and integers as well as a number of strings, lists and maps). However it is important that this not regress app startup so perhaps another approach would be warranted.

[snowp]

I think it would be interesting to see how constructing the config from the C++ config compares to the current YAML code, I don't have a great sense of which one would scale better.

Independently of this, looking into options for reducing the overall size of the config might help with startup times. Tis would make it possible to use a more ergonomic interface even if it might be slower than what we have or just a net gain if the new configuration scheme ends up being faster. What comes to mind here is using https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret#example-one-static-resource (it doesn't have to be files, can be inline) in order to define the secrets once and then refer to them by name. This avoids duplicating the long cert chains everywhere and might drastically cut down on the config processing time

[RyanTheOptimist]

Expanding on @snowp's comments based on a Slack discussion, his idea is to create an instance of the C++ class OptionsImpl which has the bootstrap config passed in via setConfigProto.

Then instead of having EnvoyEngineImpl construct a YAML config string and pass it down to JniLibrary.runEngine(), we would instead pass some "Envoy Mobile Config Builder Options" struct down to that method. Then down in C++, that code would take the builder options and transform it into the bootstrap config. Engine::main() would then need to call the Server::Options-based constructor instead of the argv-based constructor.

This seems like it could a very attractive approach.

[RyanTheOptimist]

(This is mostly a note for myself) I've read through the C++, Java, and Obj-C builders/configurations and attempted to enumerate all the different knobs we expose and which knobs apply to which languages.

All             int connect_timeout_seconds_
All             std::string stats_domain_
All             int connect_timeout_seconds_
All             int dns_refresh_seconds_
All             int dns_failure_refresh_seconds_base_
All             int dns_failure_refresh_seconds_max_
All             int dns_query_timeout_seconds_
All             std::string dns_preresolve_hostnames_
All             bool brotli_filter_
All             bool gzip_filter_
All             int h2_connection_keepalive_idle_interval_milliseconds_
All             int h2_connection_keepalive_timeout_seconds_
All             int stats_flush_seconds_
All             int stream_idle_timeout_seconds_
All             int per_try_idle_timeout_seconds_
All             std::string app_version_
All             std::string app_id_
All             std::string virtual_clusters_
All             absl::flat_hash_map<std::string, KeyValueStoreSharedPtr> key_value_stores_

C++ missing     Boolean adminInterfaceEnabled
C++ missing     Integer dnsMinRefreshSeconds
C++ missing     Boolean enableHappyEyeballs
C++ missing     Boolean enableInterfaceBinding
C++ missing     Boolean enableDrainPostDnsRefresh
C++ missing     Boolean h2ExtendKeepaliveTimeout
C++ missing     Integer maxConnectionsPerHost
C++ missing     List<EnvoyHTTPFilterFactory> httpPlatformFilterFactories
C++ missing     List<EnvoyNativeFilterConfig> nativeFilterChain
C++ missing     Map<String, EnvoyStringAccessor> stringAccessors
C++ missing     List<String> statSinks
C++ missing     TrustChainVerification trustChainVerification (enum in Java, bool in Obj-C)

Obj-C missing   bool use_system_resolver_
Obj-C missing   socket_tagging_filter_

C++ only        std::string device_os_ ("Android" in Java, "iOS" in Obj-C)

Obj-C only      BOOL forceIPv6
Obj-C only      NSString *directResponseMatchers
Obj-C only      NSString *directResponses

Java only       List<String> dnsFallbackNameservers
Java only       Boolean enableHttp3
Java only       Boolean dnsFilterUnroutableFamilies