Skip to content

Latest commit

 

History

History
408 lines (317 loc) · 13.7 KB

cert-manager.md

File metadata and controls

408 lines (317 loc) · 13.7 KB
Raw

TLS Certificate management with Cert Manager Slack

This document explains how to deploy MinIO Operator and a MinIO Tenant using certificates generated by cert-manager.

On a Kubernetes cluster, Certificate Manager requires a global level Cluster Issuer to generate intermediary Issuers and certificates. At the namespace level, Cert Manager issues certificates derived from an Issuer. The main difference between a Cluster Issuer and an Issuer is that Cluster Issuer can issue certificates for several namespaces and an Issuer can only issue them in a single namespace.

To learn more about Cert Manager Issuers refer to their documentation https://cert-manager.io/docs/concepts/issuer/.

Below is a logical view of a Kubernetes cluster:

Cert-manager-namespaces.png

This cluster contains 4 namespaces:

  • minio-operator
  • tenant-1
  • tenant-2
  • other-namespace

This guide shows you how to set up different Certificate Authorities (CA) in each namespace, all of them referencing a global Cluster Issuer.

Cert-manager Issuers.png

Cert Manager is installed in the cert-manager namespace.

The global Cluster Issuer is created in the default namespace.

A local Issuer is created in each tenant namespace.

An Issuer is also created in the minio-operator namespace. More about services that require TLS certificates in the minio-operator namespace are covered below in MinIO Operator services with cert-manager..

Note

This guide uses a self-signed Cluster Issuer. You can also use other Issuers supported by Cert Manager. The main difference is that you must provide the Issuer CA certificate to minio, instead of the CA's mentioned in this guide.

Getting Started

Prerequisites

  • Kubernetes version +v1.21. While cert-manager supports earlier K8s versions, MinIO Operator requires 1.21 or later.
  • kustomize installed
  • kubectl access to your k8s cluster

Setup Cert-manager

Install cert-manager 1.12.X LTS is preferred, or install latest, for example using kubectl.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.9/cert-manager.yaml

For more details on the Cert Manager refer to https://cert-manager.io/docs/installation/.

Create Cluster Self-signed root Issuer

The Cluster Issuer is the cluster level Issuer all other certificates are derived from. Request Cert Manager to generate this by creating a ClusterIssuer resource:

# selfsigned-root-clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-root
spec:
  selfSigned: {}
kubectl apply -f selfsigned-root-clusterissuer.yaml

MinIO Operator services with cert-manager

MinIO Operator manages the TLS certificate issuing for the services hosted in the minio-operator namespace. That is the Secure Token Service sts.

This section describes how to generate the sts TLS certificate with Cert Manager. These certificates must be issued before installing Operator. Be sure to follow step Create Cluster Self-signed root Issuer mentioned above.

Secure Token Service (STS)

MinIO STS is a service included with MinIO Operator that provides Native IAM Authentication for Kubernetes. In essence this service allows you to control access to your MinIO tenant from your kubernetes applications without having to explicitly create credentials for each application. For more information on the Service see the MinIO docs at https://min.io/docs/minio/kubernetes/upstream/developers/sts-for-operator.html. There is also an STS guide in the docs and example client code in https://github.com/minio/operator/tree/master/examples/kustomization/sts-example.

For the purpose of this guide, STS Service can be considered a webserver presented with a TLS certificate for https traffic. This guide covers how to disable the automatic generation of the certificate in MinIO Operator and issue the certificate using Cert Manager instead.

Create minio-operator namespace CA Issuer

To create the Certificate Authority (CA) certificate used to issue certificates for services in the minio-operator namespace, first create the minio-operator namespace

kubectl create ns minio-operator

Request a Certificate with spec.isCA: true specified. This is our CA for the minio-operator namespace.

# operator-ca-tls-secret.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: minio-operator-ca-certificate
  namespace: minio-operator
spec:
  isCA: true
  commonName: operator
  secretName: operator-ca-tls
  duration: 70128h # 8y
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-root
    kind: ClusterIssuer
    group: cert-manager.io
kubectl apply -f operator-ca-tls-secret.yaml

A new secret with the name operator-ca-tls is created in the minio-operator namespace, this is the CA issuing TLS certificates for the services in the minio-operator namespace.

Important

Make sure to trust this certificate in your applications that need to interact with the sts service.

Now create the Issuer:

# operator-ca-issuer.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: minio-operator-ca-issuer
  namespace: minio-operator
spec:
  ca:
    secretName: operator-ca-tls
kubectl apply -f operator-ca-issuer.yaml

Create TLS certificate for STS service

Request Cert Manager to issue a new certificate containing following DNS domains:

sts
sts.minio-operator.svc.
sts.minio-operator.svc.<cluster domain>

Important

Replace <cluster domain> with the actual values for your MinIO tenant. cluster domain is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is cluster.local, check on your coredns configuration for the correct value for your Kubernetes cluster. For example, using kubectl get configmap coredns -n kube-system -o jsonpath="{.data}". The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.)

Create a Certificate for the domains mentioned above:

# sts-tls-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sts-certmanager-cert
  namespace: minio-operator
spec:
  dnsNames:
    - sts
    - sts.minio-operator.svc
    - sts.minio-operator.svc.cluster.local
  secretName: sts-tls
  issuerRef:
    name: minio-operator-ca-issuer
kubectl apply -f sts-tls-certificate.yaml

This creates a secret called sts-tls in the minio-operator namespace.

Important

The secret name is not optional. Make sure the secret name is sts-tls by setting spec.secretName: sts-tls as in the example above.

Install Operator with Auto TLS disabled for STS

When installing the Operator deployment, make sure to set OPERATOR_STS_AUTO_TLS_ENABLED: off env variable in the minio-operator container. This prevents MinIO Operator from issuing the certificate for STS and instead wait for you to provide the TLS certificate issued by Cert Manager.

Warning

Missing to provide the secret sts-tls containing the TLS certificate or providing an invalid key-pair in the secret will prevent the STS service from start.

A way to make sure that the env variables are properly set is using kustomization to patch the minio-operator deployment:

# minio-operator/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- github.com/minio/operator/resources

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: minio-operator
      namespace: minio-operator
    spec:
      template:
        spec:
          containers:
            - name: minio-operator
              env:
                - name: OPERATOR_STS_AUTO_TLS_ENABLED
                  value: "off"
                - name: OPERATOR_STS_ENABLED
                  value: "on"
kubectl apply -k minio-operator

MinIO tenant TLS certificates with cert-manager

Create tenant-1 namespace CA Issuer

To create the Certificate Authority (CA) certificate in the namespace tenant-1, first create the namespace

kubectl create ns tenant-1

Next, request a Certificate with spec.isCA: true specified:

# tenant-1-ca-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tenant-1-ca-certificate
  namespace: tenant-1
spec:
  isCA: true
  commonName: tenant-1-ca
  secretName: tenant-1-ca-tls
  duration: 70128h # 8y
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-root
    kind: ClusterIssuer
    group: cert-manager.io
kubectl apply -f tenant-1-ca-certificate.yaml

Then create the Issuer:

# tenant-1-ca-issuer.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: tenant-1-ca-issuer
  namespace: tenant-1
spec:
  ca:
    secretName: tenant-1-ca-tls
kubectl apply -f tenant-1-ca-issuer.yaml

Deploy the tenant

Create a certificate for Tenant

Request Cert Manager issue a new TLS server certificate for MinIO that includes the following DNS domains:

minio.<namespace>
minio.<namespace>.svc
minio.<namespace>.svc.<cluster domain>
*.<tenant-name>-hl.<namespace>.svc.<cluster domain>
*.minio.<namespace>.svc.<cluster domain>
*.<tenant-name>.minio.<namespace>.svc.<cluster domain>'

Important

Replace <cluster domain> with the actual values for your MinIO tenant.

  • <cluster domain> is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is cluster.local, check on your coredns configuration for the correct value for your Kubernetes cluster. For example, using kubectl get configmap coredns -n kube-system -o jsonpath="{.data}". The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.)
  • tenant-name is the name provided to your tenant in the metadata.name of the Tenant YAML. For this example it is myminio.
  • namespace is the namespace where the tenant is created, the metadata.namespace notes that in the Tenant YAML. For this example it is tenant-1.

Create a Certificate for the domains mentioned above:

# tenant-1-minio-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tenant-certmanager-cert
  namespace: tenant-1
spec:
  dnsNames:
    - "minio.tenant-1"
    - "minio.tenant-1.svc"
    - 'minio.tenant-1.svc.cluster.local'
    - '*.minio.tenant-1.svc.cluster.local'
    - '*.myminio-hl.tenant-1.svc.cluster.local'
    - '*.myminio.minio.tenant-1.svc.cluster.local'
  secretName: myminio-tls
  issuerRef:
    name: tenant-1-ca-issuer

Tip

For this example, the Tenant name is myminio. We recommend naming the secret in the field spec.secretName as <tenant name>-tls, following the naming convention MinIO Operator uses when creates certificates with Autocert enabled (spec.requestAutoCert: true).

kubectl apply -f tenant-1-minio-certificate.yaml

Configure the tenant to use the certificate created by cert-manager

In the tenant spec, do the following:

  • Disable Autocert spec.requestAutoCert: false. This instructs Operator to not attempt to issue certificates and instead rely on Cert Manager to provide them in a secret.
  • Reference the Secret containing the TLS certificate from the previous step in spec.externalCertSecret.
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: myminio
  namespace: tenant-1
spec:
...
  ## Disable default tls certificates.
  requestAutoCert: false
  ## Use certificates generated by cert-manager.
  externalCertSecret:
    - name: myminio-tls
      type: cert-manager.io/v1
...

Trust tenant-1 CA in MinIO Operator

MinIO Operator can trust as many CA certificates as provided. To do this, create a secret with the prefix operator-ca-tls- followed by a unique identifier in the minio-operator namespace.

MinIO Operator mounts and trusts all certificates issued by the provided CA's. This is required because MinIO Operator performs health checks using the /minio/health/cluster endpoint.

If Operator is not correctly configured to trust the MinIO Certificate (or its CA), you will see an error message like the following in the Operator Pod logs:

Failed to get cluster health: Get "https://minio.tenant-1.svc.cluster.local/minio/health/cluster":
x509: certificate signed by unknown authority

For more details about health checks, see https://min.io/docs/minio/Kubernetes/upstream/operations/monitoring/healthcheck-probe.html#cluster-write-quorum.

Create operator-ca-tls-tenant-1 secret

Copy the Cert Manager generated CA public key (ca.crt) into the minio-operator namespace. This allows Operator to trust the cert-manager issued CA and the derived certificates.

Create a ca.crt file containing the CA:

kubectl get secrets -n tenant-1 tenant-1-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

Create the secret:

kubectl create secret generic operator-ca-tls-tenant-1 --from-file=ca.crt -n minio-operator

Tip

In this example we choose a secret name of operator-ca-tls-tenant-1. Note the tenant namespace tenant-1 is used as suffix for easy identification of which namespace the CA is coming from.