-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #11213 from fabriziopandini/refactor-infracluster-…
…contract 📖 Refactor InfraCluster contract
- Loading branch information
Showing
23 changed files
with
685 additions
and
367 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,89 +1,52 @@ | ||
| # Cluster Controller | ||
|
|
||
|  | ||
|
|
||
| The Cluster controller's main responsibilities are: | ||
|
|
||
| * Setting an OwnerReference on the infrastructure object referenced in `Cluster.spec.infrastructureRef`. | ||
| * Setting an OwnerReference on the control plane object referenced in `Cluster.spec.controlPlaneRef`. | ||
| * Cleanup of all owned objects so that nothing is dangling after deletion. | ||
| * Keeping the Cluster's status in sync with the infrastructureCluster's status. | ||
| * Creating a kubeconfig secret for [workload clusters](../../../reference/glossary.md#workload-cluster). | ||
|
|
||
| ## Contracts | ||
|
|
||
| ### Infrastructure Provider | ||
|
|
||
| The general expectation of an infrastructure provider is to provision the necessary infrastructure components needed to | ||
| run a Kubernetes cluster. As an example, the AWS infrastructure provider, specifically the AWSCluster reconciler, will | ||
| provision a VPC, some security groups, an ELB, a bastion instance and some other components all with AWS best practices | ||
| baked in. Once that infrastructure is provisioned and ready to be used the AWSMachine reconciler takes over and | ||
| provisions EC2 instances that will become a Kubernetes cluster through some bootstrap mechanism. | ||
|
|
||
| The cluster controller will set an OwnerReference on the infrastructureCluster. This controller should normally take no action during reconciliation until it sees the OwnerReference. | ||
|
|
||
| An infrastructureCluster controller is expected to either supply a controlPlaneEndpoint (via its own `spec.controlPlaneEndpoint` field), | ||
| or rely on `spec.controlPlaneEndpoint` in its parent [Cluster](./cluster.md) object. | ||
| The Cluster controller is responsible for reconciling the Cluster resource. | ||
|
|
||
| If an endpoint is not provided, the implementer should exit reconciliation until it sees `cluster.spec.controlPlaneEndpoint` populated. | ||
| In order to allow Cluster provisioning on different type of infrastructure, The Cluster resource references | ||
| an InfraCluster object, e.g. AWSCluster, GCPCluster etc. | ||
|
|
||
| The Cluster controller bubbles up `spec.controlPlaneEndpoint` and `status.ready` into `status.infrastructureReady` from the infrastructureCluster. | ||
| The [InfraCluster resource contract](../../providers/contracts/infra-cluster.md) defines a set of rules a provider is expected to comply in order to allow | ||
| the expected interactions with the Cluster controller. | ||
|
|
||
| #### Required `status` fields | ||
| Among those rules: | ||
| - InfraCluster SHOULD report a [controlplane endpoint](../../providers/contracts/infra-cluster.md#infracluster-control-plane-endpoint) for the Cluster | ||
| - InfraCluster SHOULD report available [failure domains](../../providers/contracts/infra-cluster.md#infracluster-failure-domains) where machines should be placed in | ||
| - InfraCluster MUST report when Cluster's infrastructure is [fully provisioned](../../providers/contracts/infra-cluster.md#infracluster-initialization-completed) | ||
| - InfraCluster SHOULD report [conditions](../../providers/contracts/infra-cluster.md#infracluster-conditions) | ||
| - InfraCluster SHOULD report [terminal failures](../../providers/contracts/infra-cluster.md#infracluster-terminal-failures) | ||
|
|
||
| The InfrastructureCluster object **must** have a `status` object. | ||
| Similarly, in order to support different solutions for control plane management, The Cluster resource references | ||
| an ControlPlane object, e.g. KubeadmControlPlane, EKSControlPlane etc. | ||
|
|
||
| The `spec` object **must** have the following fields defined: | ||
| The [ControlPlane resource contract](../../providers/contracts/control-plane.md) defines a set of rules a provider is expected to comply in order to allow | ||
| the expected interactions with the Cluster controller. | ||
|
|
||
| - `controlPlaneEndpoint` - identifies the endpoint used to connect to the target's cluster apiserver. | ||
| Considering all the info above, the Cluster controller's main responsibilities are: | ||
|
|
||
| The `status` object **must** have the following fields defined: | ||
|
|
||
| - `ready` - a boolean field that is true when the infrastructure is ready to be used. | ||
|
|
||
| #### Optional `status` fields | ||
|
|
||
| The `status` object **may** define several fields that do not affect functionality if missing: | ||
|
|
||
| * `failureReason` - is a string that explains why a fatal error has occurred, if possible. | ||
| * `failureMessage` - is a string that holds the message contained by the error. | ||
| * `failureDomains` - is a `FailureDomains` type indicating the failure domains that machines should be placed in. `FailureDomains` | ||
| is a map, defined as `map[string]FailureDomainSpec`. A unique key must be used for each `FailureDomainSpec`. | ||
| `FailureDomainSpec` is defined as: | ||
| - `controlPlane` (bool): indicates if failure domain is appropriate for running control plane instances. | ||
| - `attributes` (`map[string]string`): arbitrary attributes for users to apply to a failure domain. | ||
| * Setting an OwnerReference on the infrastructure object referenced in `Cluster.spec.infrastructureRef`. | ||
| * Setting an OwnerReference on the control plane object referenced in `Cluster.spec.controlPlaneRef`. | ||
| * Keeping the Cluster's status in sync with the InfraCluster and ControlPlane's status. | ||
| * If no ControlPlane object is referenced, create a kubeconfig secret for [workload clusters](../../../reference/glossary.md#workload-cluster). | ||
| * Cleanup of all owned objects so that nothing is dangling after deletion. | ||
|
|
||
| Note: once any of `failureReason` or `failureMessage` surface on the cluster who is referencing the infrastructureCluster object, | ||
| they cannot be restored anymore (it is considered a terminal error; the only way to recover is to delete and recreate the cluster). | ||
|  | ||
|
|
||
| Example: | ||
| ```yaml | ||
| kind: MyProviderCluster | ||
| apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3 | ||
| spec: | ||
| controlPlaneEndpoint: | ||
| host: example.com | ||
| port: 6443 | ||
| status: | ||
| ready: true | ||
| ``` | ||
| ### Kubeconfig Secrets | ||
|
|
||
| ### Secrets | ||
| In order to create a kubeconfig secret, it is required to have a certificate authority (CA) for the cluster. | ||
|
|
||
| If you are using the kubeadm bootstrap provider you do not have to provide any Cluster API secrets. It will generate | ||
| all necessary CAs (certificate authorities) for you. | ||
| all necessary CAs for you. | ||
|
|
||
| However, if you provide a CA for the cluster then Cluster API will be able to generate a kubeconfig secret. | ||
| This is useful if you have a custom CA or do not want to use the bootstrap provider's generated self-signed CA. | ||
| As alternative users can provide custom CA as described in [Using Custom Certificates](../../../tasks/certs/using-custom-certificates.md). | ||
|
|
||
| | Secret name | Field name | Content | | ||
| |:---:|:---:|:---:| | ||
| |`<cluster-name>-ca`|`tls.crt`|base64 encoded TLS certificate in PEM format| | ||
| |`<cluster-name>-ca`|`tls.key`|base64 encoded TLS private key in PEM format| | ||
|
|
||
| Alternatively can entirely bypass Cluster API generating a kubeconfig entirely if you provide a kubeconfig secret | ||
| Last option, is to entirely bypass Cluster API kubeconfig generation by providing a kubeconfig secret | ||
| formatted as described below. | ||
|
|
||
| | Secret name | Field name | Content | | ||
| |:---:|:---:|:---:| | ||
| |`<cluster-name>-kubeconfig`|`value`|base64 encoded kubeconfig| | ||
| | Secret name | Field name | Content | | ||
| |:---------------------------:|:----------:|:-------------------------:| | ||
| | `<cluster-name>-kubeconfig` | `value` | base64 encoded kubeconfig | | ||
|
|
||
| Notes: | ||
| - Also renewal of the above certificate should be taken care out of band. | ||
| - This option does not prevent from providing a cluster CA which is required also for other purposes. |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| ## Implementation best practices | ||
|
|
||
| Cluster API doesn't define strict rules about how providers should implement controllers. | ||
|
|
||
| However, some best practice are worth to notice: | ||
|
|
||
| - Infrastructure objects (e.g. load balancers, VMs etc) generated by the Infra providers SHOULD adopt a naming | ||
| convention that directly links to the Kubernetes resource that originated those objects. | ||
| Please note that in most cases external constraints might impact this decision, like e.g. | ||
| - Differences in naming conversions from Kubernetes CRDs and the target infrastructure | ||
| - The fact that the InfraCluster Kubernetes CRD is namespace-scoped while target infrastructure might have different approaches | ||
| to grouping resources | ||
|
|
||
| - Naming convention above should not be used and advertised as a contract to build on top. Instead more robust mechanism | ||
| MUST always be provided and used for identifying objects, like tagging or labeling. | ||
| Please note that this is necessary not only to prevent issues in case Cluster API changes default naming strategies | ||
| for the Kubernetes objects generated by core controllers, but also to handle use cases where users intentionally influence Cluster API naming strategies. | ||
|
|
||
| - Cluster API offers a great development environment based on Tilt, which can be easily extended to work with any provider. Use it! | ||
| See [Rapid iterative development with Tilt] | ||
|
|
||
| - Cluster API defines a set of best practices and standards that, if adopted, could speed up provider development and improve | ||
| consistency with core Cluster API. See: | ||
| - [Logging] | ||
| - [Tuning controllers] | ||
|
|
||
| - Cluster API implements a test framework that, if adopted, could help in ensuring the quality of the provider. See: | ||
| - [Testing] | ||
| - [Developing E2E tests] | ||
|
|
||
| - While standard security practices for developing Kubernetes controllers apply, it is important to recognize that | ||
| given that infrastructure provider deal with cloud credentials and cloud infrastructure, there are additional critical | ||
| security concern that must be addressed to ensure secure operations. See: | ||
| - [Infrastructure Provider Security Guidance] | ||
|
|
||
| [Rapid iterative development with Tilt]: ../core/tilt.md | ||
| [Logging]: ../core/logging.md | ||
| [Testing]: ../core/testing.md | ||
| [Developing E2E tests]: ../core/e2e.md | ||
| [Tuning controllers]: ../core/tuning.md | ||
| [Infrastructure Provider Security Guidance]: security-guidelines.md |
Oops, something went wrong.