diff --git a/site-src/geps/gep-1282.md b/site-src/geps/gep-1282.md index 89b9051fde..dd46ce31f0 100644 --- a/site-src/geps/gep-1282.md +++ b/site-src/geps/gep-1282.md @@ -36,6 +36,7 @@ The purpose of this GEP is to add APIs for capturing those backend capabilities As implementations have started to build out HTTPRoute support, requests for some common capabilities have started coming up, particularly TLS re-encryption (encryption between Gateway and backend), Websockets tracking, and HTTP/2 support. Evan Anderson opened a [discussion that generated a lot of interest](https://github.com/kubernetes-sigs/gateway-api/discussions/1244), and during that discussion, we brought up a few more points: + * Whatever we do to solve this in the HTTPRoute case may be applicable describing more general service properties, like identity, which could be useful for other Route use cases. * This may be very useful for mesh/GAMMA use cases as well as more generally for adding arbitrary future capabilities to the backend service. An example is a CA cert for connecting to the backend - that’s tightly bound to the service, but there’s nowhere to put it at the moment. @@ -46,25 +47,30 @@ This initial form of this GEP is for the Gateway API community to agree on what ### Why build something? We've got the following feature requests and discussions in the Gateway API repo: -- [#1244](https://github.com/kubernetes-sigs/gateway-api/discussions/1244) : Unclear how to specify upstream (webserver) HTTP protocol. This issue describes the problems that Evan had in trying to be able to define if a backend behind a HTTPRoute supports HTTP2 over cleartext or websockets. The question of how to tell the Gateway implementation that the backend needs TLS details for a proxy-based implementation to be able to connect it also came up in the discussion. -- [#1285](https://github.com/kubernetes-sigs/gateway-api/discussions/1285) has a more specific discussion about how different ingress implementations allow this to be configured today, whether that's with the Ingress resource or their own custom one. The great roundup that Candace did is reproduced in the next few bullet points. +* [#1244](https://github.com/kubernetes-sigs/gateway-api/discussions/1244): Unclear how to specify upstream (webserver) HTTP protocol. This issue describes the problems that Evan had in trying to be able to define if a backend behind a HTTPRoute supports HTTP2 over cleartext or websockets. The question of how to tell the Gateway implementation that the backend needs TLS details for a proxy-based implementation to be able to connect it also came up in the discussion. + +* [#1285](https://github.com/kubernetes-sigs/gateway-api/discussions/1285) has a more specific discussion about how different ingress implementations allow this to be configured today, whether that's with the Ingress resource or their own custom one. The great roundup that Candace did is reproduced in the next few bullet points. + * Istio uses a [DestinationRule resource with ClientTLSSettings](https://istio.io/latest/docs/reference/config/networking/destination-rule/#ClientTLSSettings) to capture TLS details, and the DestinationRule resource also holds traffic policy information like load balancing algorithm, connection pool size, and so on. * Openshift’s Route resource allows the [configuration of reencryption](https://docs.openshift.com/container-platform/4.10/networking/routes/secured-routes.html#nw-ingress-creating-a-reencrypt-route-with-a-custom-certificate_secured-routes) specifically, along with custom certificate details. * Contour’s HTTPProxy captures TLS details using an Envoy client certificate, destination CA certificate, and optional SubjectName which sets what Envoy should expect to see from the backend service, all inside the HTTPProxy resource. It also requires either a Protocol field inside the HTTProxy, or an annotation on the Service that tells Contour that the service expects TLS. This is [all documented](https://projectcontour.io/docs/v1.21.1/config/upstream-tls/), but I should note that Contour’s docs use the Envoy convention where a backend in Gateway parlance is called an Upstream (which may be confusing if you’re not used to it). * Linkerd uses a [Server resource](https://linkerd.io/2.11/reference/authorization-policy/#server) (which is functionally pretty similar to Service in that it associates a name with a Pod selector, but also has other details like if the service supports Proxy protocol), along with a ServerAuthorization resource that specifies some constructs that sit more at the service mesh level, including identity and access control. In terms of other implementations existing use cases for features like this: + - For Contour, there are annotations to allow the configuration of the following on a backend Service: - Max connections - Max pending requests - Max requests - Max retries - Upstream protocol: This is what allows Contour to handle switching protocols for the backend service. + - In Istio, the `DestinationRule` resource allows the configuration of many settings like this: - Load balancer algorithms - Connection pool settings - Outlier Detection - Tunnel settings for non-HTTP workloads + - In Consul, [`ServiceDefaults`](https://www.consul.io/docs/connect/config-entries/service-defaults) allows specifying similar configuration: - Max connections - Max pending requests @@ -74,6 +80,7 @@ In terms of other implementations existing use cases for features like this: - TLS SNI (we don't support configuring a specific CA at this granularity, but publishing a public key in a status field could be useful) The properties we're talking about all share two things: + - They are tightly bound to the backend, rather than being important at the route level - And the service owner (application developer in the Gateway API personas) should control these settings, not the owner of the Gateway implementation (the Cluster Operator) @@ -90,6 +97,7 @@ We’re looking to add specific, structured extension points somewhere in the re Those specific, structured extension points need to be in a place where they can be owned by the person who owns the backend, since that could be different to the person who owns the Route. That is, whatever we choose should be something that extends a thing that is referred to by a `backendRef` in a Route, not inside the Route. The initial list includes, but is not limited to: + * TLS information for connection from the Gateway to the backend service. Note that this doesn’t include any information used for service mesh encryption, just what a Gateway’s proxy would need to be able to connect to the backend service. * Websocket protocol information for the backend service. * Protocol disambiguation for “upgradeable” protocols like HTTP/2 and HTTP/1.1 which operate on the same port. (Websockets may be another case of this.) @@ -98,12 +106,292 @@ Of course, from the section above, you can see that there are many other feature ## API -To be written later, once we have agreement on the “what” and the “why”. (This is the “how”). - +A new object, BackendCapabilities, is added to the API that serves as a wrapper to a Kubernetes Service, and provisions +backend capabilities for a Service that requires them. As discussed by the Gateway API maintainers and community, this +architectural choice is made because we would ideally like to locate this capabilities set within the Service but until +that is possible in Kubernetes, we can start with a wrapper to the Service. + +BackendCapabilities contains a BackendCapabilitiesSpec object that specifies the targeted backend and defines each of +the different backend capabilities. In this GEP, TLS encryption between gateway and backend is proposed as a capability, +but with a design that allows additional capabilities to be added as needed in subsequent versions of +Gateway API. BackendCapabilities also contains a BackendCapabilitiesStatus which holds the current state. + +In https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1beta1/shared_types.go add a new type: +```go +// BackendCapabilities references a Kubernetes Service and defines any backend +// capabilities that Service requires. +// +// +optional +type BackendCapabilities struct { + // Name is the name of this backend capability set. + // +required + Name string `json:”name”` + + // Spec specifies the capabilities supported by this BackendCapabilities + // Service wrapper, if any. The capabilities are predefined, and have two + // characteristics in common: they are tightly bound to the backend, + // rather than being important at the route level, and the service owner + // (application developer in the Gateway API personas) should control + // these settings, not the owner of the Gateway implementation (the + // Cluster Operator in the Gateway API personas). + // + // Support: Extended + // + // +required + Spec BackendCapabilitiesSpec `json:”spec”` + + // Status represents the current state of the backend and its capabilities. + // +optional + Status *BackendCapabilitiesStatus `json:”status,omitempty”` +} + +// BackendTargetReference identifies an API object to apply BackendCapabilities to. +// +type BackendTargetReference struct { + // Group is the group of the target resource. + Group Group `json:"group"` + + // Kind is kind of the target resource. + // Valid values include: + // * "Service" (the default if no Kind is specified) + // * "ServiceImport" + // + // Invalid values include: + // + // * "invalid/kind" - "/" is an invalid character + // +required + Kind string `json:"kind"` + + // Name is the name of the target resource. + Name ObjectName `json:"name"` +} + +// BackendCapabilitiesSpec defines a group of specific properties held by its +// affiliated backend, and specified by the backend owner (typically the +// application developer). +// +// Examples of properties that belong in the BackendCapabilitiesSpec include: +// TLS re-encryption, session state, timeouts, retries, Websockets, HTTP/2, +// load balancer algorithm, healthcheck info, external traffic policy, +// connection limits, error page, protocol switch, alerts, and others. +// +// Support: Extended +// +type BackendCapabilitiesSpec struct { + // Backend refers to the object to which we apply these capabilities + // +required + Backend backendTargetReference `json:"backend,omitempty"` + + // TLS is the section of details on encryption processing in the traffic + // hop from Gateway to backend, sometimes known as upstream TLS or re-encrypt. + // A xRoute configured as passthrough cannot specify a TLS encrypt type. + // TLS is useful for HTTPRoute and should not be used on UDP/TCP/TLSRoute. + // + // +optional + TLS *EncryptType `json:"tls,omitempty"` +} + +// EncryptType describes the details needed by the client (Gateway) to implement +// encryption from the client to the destination endpoint. +type EncryptType struct { + // The minimal TLSVersion that is accepted by the endpoint. + // If empty, there is no minimal backend TLSVersion. + // +optional + MinVersion *TLSVersion `json:"minVersion,omitempty"` + + // The maximum TLSVersion that is accepted by the endpoint. + // If empty, there is no maximum backend TLSVersion. + // +optional + MaxVersion *TLSVersion `json:"maxVersion,omitempty"` + + // CipherSuites is the minimum cipher suite that the backend accepts. + // If empty, anything that is appropriate for the MinVersion is accepted. + // If the MinVersion is not specified, the result is implementation + // dependent, e.g. based on the minimimal TLS version supported by the + // implementation. + // +optional + CipherSuites []string `json:"cipherSuites,omitempty"` + + // CACerts is the PEM bundle containing Certificate Authority, trusted by both + // client and endpoint, and intermediate certificates needed to complete the + // certificate chain of trust. If empty, the system root certs should be used. + // + // +optional + CACerts *SecretObjectReference `json:"caCerts,omitempty"` + + // ServiceNames provide multiple named domains which will share a certificate + // for the endpoint. + // +optional + ServiceNames []string `json:"serviceNames,omitempty"` +} + +// BackendCapabilitiesStatus describes the status of the backend and its +// capabilities. +type backendCapabilitiesStatus struct { + // List of conditions that represent the current status of the + // the backend capabilities object. + // TODO: Define constants to represent typical status conditions + // + // +kubebuilder:validation:MinItems=1 + // +kubebuilder:validation:MaxItems=8 + Conditions []metav1.Condition `json:”conditions,omitempty"` +} +``` + +###Example Capabilities +For a Service `private-service` defined as: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: private-service +spec: + selector: + app.kubernetes.io/name: privateApp +ports: + - protocol: TCP + port: 80 + targetPort: 9099 +``` + +We could create a backendCapabilities for TLS re-encrypt on `private-service` defined as: + +```yaml +apiversion: v1beta1 +kind: backendCapabilities +metadata: + name: private-service-caps +spec: + backend: + kind: Service + name: private-service + TLS: + minVersion: 1.3 + cipherSuites: + - TLS_AES_256_GCM_SHA384 + - TLS_CHACHA20_POLY1305_SHA256 + - TLS_AES_128_GCM_SHA256 + caCerts: my-cert-bundle + serviceNames: + - private-service.example.com + - dev.example.com +``` + +Implementers of TLS re-encrypt would have to check for backendCapabilities that shared the same spec.backend.name as +the targeted Service, to discover the capabilities specification they would need to perform encryption between the +Gateway and the backend. ## Alternatives -Again, we need to wait until we’re writing the “how”. +### Alternative 1 - map of structs + +One alternative merits a mention. In this alternative proposal, the API will be composed of several new objects: +BackendObjectCapabilities, an instance of BackendObjectCapabilitiesMap, defined as a mapping of capability name to an +array of BackendObjectCapability objects. BackendObjectCapability objects are simple but expressive Group-Kind-Name +objects with an additional Value field implemented as an interface so that it can store any type of data. + +In https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1beta1/shared_types.go: +```go +// BackendRef defines how a Route should forward a request to a Kubernetes +// resource. +// +// ... +type BackendRef struct { + // BackendObjectReference references a Kubernetes object. + BackendObjectReference `json:",inline"` + + // BackendObjectCapabilities specifies the capabilities supported by this + // BackendRef, if any. The capabilities are not predefined, but have two + // characteristics in common: they are tightly bound to the backend, + // rather than being important at the route level, and the service owner // (application developer in the Gateway API personas) should control + // these settings, not the owner of the Gateway implementation (the + // Cluster Operator in the Gateway API personas). + // + // There are a maximum of 32 BackendObjectCapabilities per BackendRef. + // + // Support: Extended + // + // +optional + // +kubebuilder:validation:MaxItems=32 + BackendObjectCapabilities *BackendObjectCapabilitiesMap `json:”backendObjectCapabilitiesMap,omitempty”` + + // Weight specifies the proportion of requests forwarded to the referenced + //... + Weight *int32 `json:"weight,omitempty"` +} + +// BackendObjectCapabilitiesMap defines a mapping of BackendObjectCapability name +// to an array of BackendObjectCapability +BackendObjectCapabilitiesMap map[string][]BackendObjectCapability + +// BackendObjectCapability defines a capability or property held by its +// affiliated BackendObject, and specified by the BackendObject owner +// (typically the application developer). Examples of BackendObjectCapability +// use cases include: TLS re-encryption, session state, timeouts, retries, +// Websockets, HTTP/2, load balancer algorithm, healthcheck info, external +// traffic policy, connection limits, error page, protocol switch, alerts. +// +// The BackendObject must be valid in the cluster; the Group and Kind must +// be registered in the cluster for this reference to be valid. +// +// Support: Extended +// +// References to objects with invalid Group and Kind are not valid. +type BackendObjectCapability struct { + // Group is the group of the referent. + // + // Support: Core + // + // +kubebuilder:default=gateway.networking.k8s.io + // +optional + Group *Group `json:"group,omitempty"` + + // Kind is kind of the referent. + // + // Support: Custom + // + // +kubebuilder:default=Gateway + // +optional + Kind *Kind `json:"kind,omitempty"` + + // Name is the name of the referent. + // + // Support: Core + Name ObjectName `json:"name"` + + // Value is the value of the referent. The type is not specifically + // defined and is left up to the implementation to infer. + // + // Support: Extended + // + // +optional + Value interface{} `json:”value”` +} + +``` +### Example Capabilities for Alternative 1 + +TLS information for connection from the Gateway to the backend service. What a Gateway’s proxy would need to be able to +connect to the backend service. + +The BackendCapabilities could be defined by an implementation and look something like: + +```yaml + +backendCapabilities: + tls-reencrypt: + - name: version + value: 1.3 + - name: clientCert + value: clientSecret + - name: privateKey + value: pkeySecret + - name: caCerts + value: cacertSecret + - name: backendCert + value: backendSecret +``` ## References