[KEP] Context Updates for KIC 2.0 KEP

railgun/keps/0003-kic-kubebuilder-re-architecture.md

@@ -26,6 +26,7 @@ For reference see the milestones related to this proposal to check the progress
 - [Design Details](#design-details)
   - [Test Plan](#test-plan)
   - [Graduation Criteria](#graduation-criteria)
+  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
 - [Implementation History](#implementation-history)
 - [Alternatives](#alternatives)
 <!-- /toc -->
@@ -64,6 +65,10 @@ Historically the [Kong Kubernetes Ingress Controller (KIC)][kic] was built on ol
 
 ### Non-Goals
 
+#### Configuration Monolith Re-architecture
+
+When we first started experimenting and prototyping for this KEP we were motivated to deconstruct the existing monolithic controller such that individual controllers for APIs could be separate microservices and autonomous. Due to limitations of maintainer capacity and some desires for upstream changes that would not be able to occur logistically fast enough to support us (namely having a single upstream API to develop against rather than a separate API for DB vs DBLESS Kong instances) we've pulled this work out of scope. We are however still motivated to make this change and continue our re-architecture, just consider it out of scope for this KEP and it will become the subject of its own.
+
 #### Kubebuilder Controller Management
 
 Despite the motivation present in this KEP to automate some of our controller management, logistics and time constraints have led us to keeping the conversion our existing controller to `kubebuilder` managed controllers _out of scope_ for this KEP. For this iteration we will focus on using the API, CRD, and configuration management features of `kubebuilder`, but the controller management features will be considered as part of a later iteration to reduce the number of changes we make with a single release (we will however still convert to controller runtime and ultimately use the kubebuilder provided controller machinery to replace our historical machinery).
@@ -125,6 +130,76 @@ As a contributor to the KIC I want to be able to quickly contribute new ideas an
 
 As a user of KIC, I want to be able to inspect the intermediate objects produced by KIC (collected KongState, generated decK config) for debugging purposes, as inspired by [this review comment](https://github.com/Kong/kubernetes-ingress-controller/pull/991#pullrequestreview-570627606).
 
+## Design Details
+
+This re-architecture is focused on moving the existing APIs, controllers, and libraries onto [Kubebuilder SDK][kb] for multiple expected benefits including (but not limited to):
+
+- reducing large amounts of code, particularly by using [controller-runtime][cr] to replace our own machinery in several places
+- configuring our APIs within the Kubebuilder SDK, making management (creation, update, deletion) of APIs more automated
+- automating management and build of our manifests, including controller manager, CRDs, RBAC configurations, e.t.c.
+- generating our public Go API of `kubernetes.ClientSet` via [client-gen][cg]
+
+These among other gains will bring us closer to how upstream works and will reduce the amount of maintenance we have to perform to keep up to date and add features.
+
+On top of transplanting our APIs and adding a new Go API, we also want to transplant the existing monolithic controller onto `controller-runtime` and align ourselves with other controller implementations throughout the community.
+
+[kb]:https://github.com/kubernetes-sigs/kubebuilder
+[cr]:https://github.com/kubernetes-sigs/controller-runtime
+[cg]:https://github.com/kubernetes/code-generator
+
+### Test Plan
+
+Prior to these efforts only minimal testing for the controller and the API functionality existed, with these efforts we will add a new integration test suite that covers:
+
+- significantly improve our unit testing coverage
+- establish integration tests in Golang which test all our APIs against real Kubernetes clusters
+- document testing requirements for new contributions going forward in CONTRIBUTING.md
+
+**NOTE**: Testing of our new Go API is covered for free by `client-gen`.
+
+#### TCPIngress Example
+
+The following is an example testing plan that outlines how we will test each of our APIs as part of this effort.
+
+##### Validation Tests
+
+- [ ] several valid permutations of each available field in the `TCPIngress` API spec are covered
+- [ ] several invalid permutations of each available field in the `TCPIngress` API spec are covered
+
+##### General Functionality Tests
+
+- [ ] `TCPIngress` resources can be created and route properly to the relevant `Service`, the route becomes available in under 1 minute
+- [ ] `TCPIngress` resources can be updated for a new backend `Service`, the routes change properly in under 1 minute
+- [ ] When `TCPIngress` spec changes, the behavior of persistent connections is tested (backend specific)
+- [ ] When `TCPIngress` spec changes, repeated parallel transient connections to the backend show no failures.
+- [ ] Long running persistent connections (where the backend server doesn't support reconnect mechanisms) disconnect and fail properly when pods change
+- [ ] `TCPIngress` resources can be deleted and the backend route is disconnected in under 1 minute
+
+##### Status Tests
+
+- [ ] `TCPIngress` status is checked for consistent `LoadBalancer` state between creation and multiple updates to the spec
+
+##### Specific Functionality Tests
+
+- [ ] `TCPIngress` ingress TLS configuration
+- [ ] `TCPIngress` ingress TLS configuration for multiple hosts
+
+##### Performance Tests
+
+- [ ] Multiple `TCPIngress` resources can be rapidly created, each backend resolving in under 1 minute
+- [ ] Bulk parallel connections through `TCPIngress` succeed, no failures at KIC/Kong level
+- [ ] Multiple `TCPIngress` resources can be rapidly deleted, each backend route becomes disconnected in under 1 minute
+
+### Graduation Criteria
+
+- [x] all APIs are ported to the new architecture
+- [x] all APIs are exposed via the public Go API
+- [x] all APIs are given a controller responsible for ingest of the relevant resources
+- [ ] the existing controller that configures the Kong Admin API is ported
+- [ ] all APIs receive new automated testing equivalent to the example provided in the above `Testing Plan` section
+- [ ] an alpha release cycle with feedback and bug reporting is conducted
+- [ ] an beta release cycle with feedback and bug reporting is conducted maintaining backwards compatibility
+
 ## Implementation History
 
 - spike and prototype started to re-architect the KIC and move it to modern tooling: https://github.com/kong/railgun
@@ -150,4 +225,10 @@ As a user of KIC, I want to be able to inspect the intermediate objects produced
 
 ## Alternatives
 
+### CRD/Secret vs. In-Memory Cache
+
+To help break apart the monolithic controller from 1.x we considered using a `Secret` or `CRD` as the caching location for resources as an interim solution between previous arch and the future arch we wanted to define, however the timing and logistics of that simply didn't work for this KEPs scope and limitations such as maximum object size for `Secrets` led us to stop on this and save it for a later iteration.
+
+### OperatorSDK vs. Kubebuilder
+
 The [OperatorSDK][osdk] from [Redhat][rhel] was considered for our new Kubernetes SDK, but ultimately decided against due to lack of familiarity and preferring a more generic and flexible toolkit.