Skip to content
This repository has been archived by the owner on Jun 29, 2022. It is now read-only.

Commit

Permalink
Add ability to provide extra kube-apiserver flags
Browse files Browse the repository at this point in the history 
This commit adds an ability to provide extra flags for self-hosted
kube-apiserver, so this feature can be used when configuring dex and
gangway for OIDC authentication to the cluster, as currently any manual
changes to kube-apiserver pods will be overwritten during cluster
upgrade.

Closes #122

Signed-off-by: Mateusz Gozdek <mateusz@kinvolk.io>
  • Loading branch information
@invidian
invidian committed Mar 18, 2020
1 parent cfacb2d commit 6e2e567
Show file tree
Hide file tree
Showing 25 changed files with 244 additions and 164 deletions.
22 changes: 11 additions & 11 deletions assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/bootkube.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@
module "bootkube" {
source = "../../../bootkube"

cluster_name = var.cluster_name
api_servers = [format("%s.%s", var.cluster_name, var.dns_zone)]
etcd_servers = aws_route53_record.etcds.*.fqdn
asset_dir = var.asset_dir
network_mtu = var.network_mtu
pod_cidr = var.pod_cidr
service_cidr = var.service_cidr
cluster_domain_suffix = var.cluster_domain_suffix
enable_reporting = var.enable_reporting
enable_aggregation = var.enable_aggregation

cluster_name = var.cluster_name
api_servers = [format("%s.%s", var.cluster_name, var.dns_zone)]
etcd_servers = aws_route53_record.etcds.*.fqdn
asset_dir = var.asset_dir
network_mtu = var.network_mtu
pod_cidr = var.pod_cidr
service_cidr = var.service_cidr
cluster_domain_suffix = var.cluster_domain_suffix
enable_reporting = var.enable_reporting
enable_aggregation = var.enable_aggregation
kube_apiserver_extra_flags = var.kube_apiserver_extra_flags
certs_validity_period_hours = var.certs_validity_period_hours
}
6 changes: 6 additions & 0 deletions assets/lokomotive-kubernetes/aws/flatcar-linux/kubernetes/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -182,3 +182,9 @@ variable "certs_validity_period_hours" {
type = number
default = 8760
}

variable "kube_apiserver_extra_flags" {
description = "Extra flags passed to self-hosted kube-apiserver."
type = list(string)
default = []
}
1 change: 1 addition & 0 deletions assets/lokomotive-kubernetes/bare-metal/flatcar-linux/kubernetes/bootkube.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ module "bootkube" {
cluster_domain_suffix = var.cluster_domain_suffix
enable_reporting = var.enable_reporting
enable_aggregation = var.enable_aggregation
kube_apiserver_extra_flags = var.kube_apiserver_extra_flags

certs_validity_period_hours = var.certs_validity_period_hours
}
6 changes: 6 additions & 0 deletions assets/lokomotive-kubernetes/bare-metal/flatcar-linux/kubernetes/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -162,3 +162,9 @@ variable "certs_validity_period_hours" {
type = number
default = 8760
}

variable "kube_apiserver_extra_flags" {
description = "Extra flags passed to self-hosted kube-apiserver."
type = list(string)
default = []
}
1 change: 1 addition & 0 deletions assets/lokomotive-kubernetes/bootkube/assets.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ resource "local_file" "kube-apiserver" {
aggregation_client_key = var.enable_aggregation == true ? base64encode(join(" ", tls_private_key.aggregation-client.*.private_key_pem)) : ""
replicas = length(var.etcd_servers)
expose_on_all_interfaces = var.expose_on_all_interfaces
extra_flags = var.kube_apiserver_extra_flags
})
}

Expand Down
6 changes: 6 additions & 0 deletions assets/lokomotive-kubernetes/bootkube/resources/charts/kube-apiserver.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,9 @@ apiserver:
trustedCertsDir: ${trusted_certs_dir}
replicas: ${replicas}
exposeOnAllInterfaces: ${expose_on_all_interfaces}
%{~ if length(extra_flags) > 0 ~}
extraFlags:
%{~ for f in extra_flags ~}
- ${f}
%{~ endfor ~}
%{~ endif ~}
6 changes: 6 additions & 0 deletions ...omotive-kubernetes/bootkube/resources/charts/kube-apiserver/templates/kube-apiserver.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,9 @@ spec:
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
{{ end -}}
{{- range .Values.apiserver.extraFlags -}}
{{ . }} \
{{- end }}
--storage-backend=etcd3
{{- else }}
- /hyperkube
Expand Down Expand Up @@ -141,6 +144,9 @@ spec:
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
{{- end }}
{{- range .Values.apiserver.extraFlags -}}
- {{ . }}
{{- end }}
readinessProbe:
tcpSocket:
port: 6443
Expand Down
1 change: 1 addition & 0 deletions assets/lokomotive-kubernetes/bootkube/resources/charts/kube-apiserver/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ apiserver:
# service (kubernetes.default.svc) and on all interfaces on port 6443.
# If false, it will be exposed only on HostIP on port 6443.
exposeOnAllInterfaces: false
extraFlags: []
6 changes: 6 additions & 0 deletions assets/lokomotive-kubernetes/bootkube/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,3 +134,9 @@ variable "expose_on_all_interfaces" {
type = bool
default = false
}

variable "kube_apiserver_extra_flags" {
description = "Extra flags passed to self-hosted kube-apiserver."
type = list(string)
default = []
}
2 changes: 2 additions & 0 deletions assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/bootkube.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,6 @@ module "bootkube" {
container_arch = var.os_arch

expose_on_all_interfaces = true

kube_apiserver_extra_flags = var.kube_apiserver_extra_flags
}
6 changes: 6 additions & 0 deletions assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -175,3 +175,9 @@ variable "tags" {
type = list(string)
default = ["ManagedBy:Lokomotive", "CreatedBy:Unspecified"]
}

variable "kube_apiserver_extra_flags" {
description = "Extra flags passed to self-hosted kube-apiserver."
type = list(string)
default = []
}
5 changes: 5 additions & 0 deletions ci/aws/aws-cluster.lokocfg.envsubst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@ cluster "aws" {
"deployment" = "ci"
}
}

# Smoke test for extra flags feature.
kube_apiserver_extra_flags = [
"-v=1",
]
}

component "openebs-operator" {}
Expand Down
5 changes: 5 additions & 0 deletions ci/baremetal/baremetal-cluster.lokocfg.envsubst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,9 @@ cluster "bare-metal" {
"node2",
"node3",
]

# Smoke test for extra flags feature.
kube_apiserver_extra_flags = [
"-v=1",
]
}
5 changes: 5 additions & 0 deletions ci/packet/packet-cluster.lokocfg.envsubst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ cluster "packet" {
node_type = "c2.medium.x86"
labels = "testing.io=yes,roleofnode=testing"
}

# Smoke test for extra flags feature.
kube_apiserver_extra_flags = [
"-v=1",
]
}

component "openebs-operator" {}
Expand Down
4 changes: 4 additions & 0 deletions docs/configuration-reference/platforms/aws.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ variable "worker_disk_type" {}
variable "worker_disk_iops" {}
variable "worker_spot_price" {}
variable "target_groups" {}
variable "kube_apiserver_extra_flags" {}
backend "s3" {
bucket = var.state_s3_bucket
Expand Down Expand Up @@ -140,6 +141,8 @@ cluster "aws" {
"key" = "value"
}
}
kube_apiserver_extra_flags = var.kube_apiserver_extra_flags
}
```

Expand Down Expand Up @@ -194,6 +197,7 @@ worker_pool "my-worker-pool" {
| `cluster_domain_suffix` | Cluster's DNS domain. | "cluster.local" | false |
| `enable_reporting` | Enables usage or analytics reporting to upstream. | false | false |
| `certs_validity_period_hours` | Validity of all the certificates in hours. | 8760 | false |
| `kube_apiserver_extra_flags` | Extra flags passed to self-hosted kube-apiserver. | [] | false |
| `worker_pool` | Configuration block for worker pools. There can be more than one. **NOTE**: worker pool name must be unique per DNS zone and region. | - | true |
| `worker_pool.count` | Number of workers in the worker pool. Can be changed afterwards to add or delete workers. | - | true |
| `worker_pool.instance_type` | AWS instance type for worker nodes. | "t3.small" | false |
Expand Down
44 changes: 24 additions & 20 deletions docs/configuration-reference/platforms/baremetal.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ variable "management_cidrs" {}
variable "node_private_cidr" {}
variable "state_s3_bucket" {}
variable "lock_dynamodb_table" {}
variable "kube_apiserver_extra_flags" {}
cluster "bare-metal" {
asset_dir = var.asset_dir
Expand Down Expand Up @@ -83,6 +84,8 @@ cluster "bare-metal" {
os_version = "current"
os_channel = "flatcar-stable"
kube_apiserver_extra_flags = var.kube_apiserver_extra_flags
}
```

Expand Down Expand Up @@ -111,26 +114,27 @@ os_version = var.custom_default_os_version

## Attribute reference

| Argument | Description | Default | Required |
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------:|:--------:|
| `asset_dir` | Location where Lokomotive stores cluster assets. | - | true |
| `cached_install` | Whether the operating system should PXE boot and install from matchbox /assets cache. Note that the admin must have downloaded the `os_version` into matchbox assets. | "false" | false |
| `cluster_name` | Name of the cluster. | - | true |
| `controller_domains` | Ordered list of controller FQDNs. Example: ["node1.example.com"] | - | true |
| `controller_macs` | Ordered list of controller identifying MAC addresses. Example: ["52:54:00:a1:9c:ae"] | - | true |
| `controller_names` | Ordered list of controller names. Example: ["node1"] | - | true |
| `k8s_domain_name` | Controller DNS name which resolves to a controller instance. Workers and kubeconfig's will communicate with this endpoint. Example: "cluster.example.com" | - | true |
| `matchbox_ca_path` | Path to the CA to verify and authenticate client certificates. | - | true |
| `matchbox_client_cert_path` | Path to the server TLS certificate file. | - | true |
| `matchbox_client_key_path` | Path to the server TLS key file. | - | true |
| `matchbox_endpoint` | Matchbox API endpoint. | - | true |
| `matchbox_http_endpoint` | Matchbox HTTP read-only endpoint. Example: "http://matchbox.example.com:8080" | - | true |
| `worker_names` | Ordered list of worker names. Example: ["node2", "node3"] | - | true |
| `worker_macs` | Ordered list of worker identifying MAC addresses. Example ["52:54:00:b2:2f:86", "52:54:00:c3:61:77"] | - | true |
| `worker_domains` | Ordered list of worker FQDNs. Example ["node2.example.com", "node3.example.com"] | - | true |
| `ssh_pubkeys` | SSH public keys for user `core`. | - | true |
| `os_version` | Flatcar Container Linux version to install. Version such as "2303.3.1" or "current". | "current" | false |
| `os_channel` | Flatcar Container Linux channel to install from ("flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"). | "flatcar-stable" | false |
| Argument | Description | Default | Required |
|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------:|:--------:|
| `asset_dir` | Location where Lokomotive stores cluster assets. | - | true |
| `cached_install` | Whether the operating system should PXE boot and install from matchbox /assets cache. Note that the admin must have downloaded the `os_version` into matchbox assets. | "false" | false |
| `cluster_name` | Name of the cluster. | - | true |
| `controller_domains` | Ordered list of controller FQDNs. Example: ["node1.example.com"] | - | true |
| `controller_macs` | Ordered list of controller identifying MAC addresses. Example: ["52:54:00:a1:9c:ae"] | - | true |
| `controller_names` | Ordered list of controller names. Example: ["node1"] | - | true |
| `k8s_domain_name` | Controller DNS name which resolves to a controller instance. Workers and kubeconfig's will communicate with this endpoint. Example: "cluster.example.com" | - | true |
| `matchbox_ca_path` | Path to the CA to verify and authenticate client certificates. | - | true |
| `matchbox_client_cert_path` | Path to the server TLS certificate file. | - | true |
| `matchbox_client_key_path` | Path to the server TLS key file. | - | true |
| `matchbox_endpoint` | Matchbox API endpoint. | - | true |
| `matchbox_http_endpoint` | Matchbox HTTP read-only endpoint. Example: "http://matchbox.example.com:8080" | - | true |
| `worker_names` | Ordered list of worker names. Example: ["node2", "node3"] | - | true |
| `worker_macs` | Ordered list of worker identifying MAC addresses. Example ["52:54:00:b2:2f:86", "52:54:00:c3:61:77"] | - | true |
| `worker_domains` | Ordered list of worker FQDNs. Example ["node2.example.com", "node3.example.com"] | - | true |
| `ssh_pubkeys` | SSH public keys for user `core`. | - | true |
| `os_version` | Flatcar Container Linux version to install. Version such as "2303.3.1" or "current". | "current" | false |
| `os_channel` | Flatcar Container Linux channel to install from ("flatcar-stable", "flatcar-beta", "flatcar-alpha", "flatcar-edge"). | "flatcar-stable" | false |
| `kube_apiserver_extra_flags` | Extra flags passed to self-hosted kube-apiserver. | [] | false |

## Applying

Expand Down
4 changes: 4 additions & 0 deletions docs/configuration-reference/platforms/packet.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ variable "management_cidrs" {}
variable "node_private_cidr" {}
variable "state_s3_bucket" {}
variable "lock_dynamodb_table" {}
variable "kube_apiserver_extra_flags" {}
backend "s3" {
bucket = var.state_s3_bucket
Expand Down Expand Up @@ -122,6 +123,8 @@ cluster "packet" {
certs_validity_period_hours = 8760
kube_apiserver_extra_flags = var.kube_apiserver_extra_flags
worker_pool "worker-pool-1" {
count = var.workers_count
Expand Down Expand Up @@ -210,6 +213,7 @@ node_type = var.custom_default_worker_type
| `reservation_ids` | Specify Packet hardware reservation ID for instances. | - | false |
| `reservation_ids_default` | Default reservation ID for nodes not listed in the `reservation_ids`. The value`next-available` will choose any reservation that matches the pool's device type and facility. | "" | false |
| `certs_validity_period_hours` | Validity of all the certificates in hours. | 8760 | false |
| `kube_apiserver_extra_flags` | Extra flags passed to self-hosted kube-apiserver. | [] | false |
| `worker_pool` | Configuration block for worker pools. There can be more than one. | - | true |
| `worker_pool.count` | Number of workers in the worker pool. Can be changed afterwards to add or delete workers. | 1 | true |
| `worker_pool.disable_bgp` | Disable BGP on nodes. Nodes won't be able to connect to Packet BGP peers. | false | false |
Expand Down
Loading

0 comments on commit 6e2e567

Please sign in to comment.