Use of Let's Encrypt wildcard certs in Kubernetes
A wildcard certificate can secure any number of subdomains of a base domain (e.g. *.example.com). This allows to use a single certificate and key pair for a domain and all of its subdomains, which can make HTTPS deployment significantly easier.
Let's Encrypt wildcard certificates support went live in March 2018.
In this blog post you will learn how to setup Kubernetes Ingress controller with Heptio Contour, automate the management and issuance of wildcard TLS certificates with Jetstack Cert-Manager and sync the TLS certs across different namespaces with AppsCode Kubed.
Pre-Requisites
You need to have Kubernetes cluster available, if you don't have it, follow one the docs below to set it up on:
Helm is installed in your Kubernetes cluster.
Contour Ingress Controller
Ok, first we need to install Kubernetes Ingress controller.
As there is no Helm chart for Heptio Contour, I wrote the chart and stored it in my Helm repository.
To be able to use it you need to add my Chart repo to Helm:
$ helm repo add rimusz https://helm-charts.rimusz.net
$ helm repo up
Ok, let's install contour chart:
helm install --name contour rimusz/contour
And to check that ingress controller is running:
$ kubectl --namespace=heptio-contour get pods -l "app=contour"
NAME READY STATUS RESTARTS AGE
contour-5d7f6fc8bd-nv474 2/2 Running 0 20s
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status with:
$ kubectl get svc --namespace heptio-contour -w contour
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
contour LoadBalancer 10.64.12.134 <pending> 80:30321/TCP,443:32400/TCP 34s
contour LoadBalancer 10.64.12.134 35.238.152.138 80:30321/TCP,443:32400/TCP 46s
Now please update your domain DNS record with the External IP.
Nice, we got our Ingress Controller installed.
TLS Cert-Manager
We are going to install cert-manager from Jetstack repo:
$ helm repo add jetstack https://charts.jetstack.io
$ kubectl apply \
-f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml
$ helm install --name cert-manager --namespace cert-manager stable/cert-manager
And we check it is running:
$ kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-5f8db6f6c4-jjzjc 1/1 Running 0 22s
cert-manager-webhook-85dd96d87-jsfjk 1/1 Running 0 22s
cert-manager-webhook-ca-sync-vwt57 0/1 Completed 0 22s
Awesome.
Generating wildcard certs with Cert-Manager
For this blog post I'm using CloudFlare (sorry I'm a big fan of their services) as DNS01 Challenge Provider, check for other supported ones here.
Now we need to create a secret with CloudFlare Global API Key, Cert-Manager Issuer with DNS1 Challenge Provider, which will use that secret and the Cert-Manager Certificate which will save the wildcard cert of *.example.com to secret example-com-tls:
$ cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-key
namespace: cert-manager
type: Opaque
data:
api-key.txt: <copy here base64 encoded CloudFlare API Key>
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: user@example.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# ACME DNS-01 provider configurations
dns01:
# Here we define a list of DNS-01 providers that can solve DNS challenges
providers:
- name: cf-dns
cloudflare:
email: user@example.com
# A secretKeyRef to a cloudflare api key
apiKeySecretRef:
name: cloudflare-api-key
key: api-key.txt
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: example-com
namespace: cert-manager
spec:
secretName: example-com-tls
issuerRef:
name: letsencrypt-staging
commonName: '*.example.com'
acme:
config:
- dns01:
provider: cf-dns
domains:
- '*.example.com'
EOF
Note: Please update the example template above with your CloudFlare API key, email address and domain name.
If all went successfully you should see your wildcard cert example-com-tls:
$ kubectl -n cert-manager get secret
NAME TYPE DATA AGE
cert-manager-cert-manager-token-whctf kubernetes.io/service-account-token 3 4h
cloudflare-api-key Opaque 1 17m
default-token-cfmln kubernetes.io/service-account-token 3 4h
letsencrypt-staging Opaque 1 37m
example-com-tls kubernetes.io/tls 2 12m
Yay.
Sync wildcard certs between Kubernetes namespaces
Kubernetes best practices recommend to install app per namespace, but our wildcard cert example-com-tls is stored in the cert-manager namespace, so we need to sync it across namespaces when the cert gets reissued or new app in the new namespace gets added.
For that we are going to use a nice tool from AppsCode Kubed.
Check their other tools, you could find more useful ones for you.
Kubed can be installed via Helm using the chart from AppsCode Charts Repository:
$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update
$ helm install appscode/kubed --name kubed --namespace kube-system \
--set apiserver.enabled=false \
--set config.clusterName=my_staging_cluster
Note: Set cluster-name to something meaningful to you, say, prod, prod-us-east, qa, etc.
Which also enables kubed's ConfigSyncer feature.
To verify that Kubed has started, run:
$ kubectl --namespace=kube-system get deployments -l "release=kubed, app=kubed"
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kubed-kubed 1 1 1 1 15s
Cool.
Ok, now let's create a few namespaces where we are going to sync our wildcard cert to:
$ kubectl create namespace demo1
$ kubectl create namespace demo2
Now we need to put the annotation as per docs to our wildcard secret, so kubed knows what to sync across namespaces:
$ kubectl annotate secret example-com-tls -n cert-manager kubed.appscode.com/sync="app=kubed"
secret "example-com-tls" annotated
From now on kubed will start syncing secret example-com-tls to only namespaces which have label-selector app=kubed.
Let's annotate namespace demo1:
$ kubectl label namespace demo1 app=kubed
Now we should be able to see the secret in the namespace demo1:
$ kubectl -n demo1 get secret example-com-tls
NAME TYPE DATA AGE
example-com-tls kubernetes.io/tls 2 2h
And if we check namespace demo2 we will not see the example-com-tls there:
$ kubectl -n demo2 get secret example-com-tls
Error from server (NotFound): secrets "example-com-tls" not found
Putting the annotation on to demo2 namespace will trigger an instant secret sync as well.
Now you can install web apps into demo1 and demo2 namespaces, and apps be able to use the same wildcard cert for their ingress rules.
What's next
Moving forward to production releases and receiving the real TLS certs from Let's Encrypt add new Issuer and Certificate as per example below:
$ cat <<EOF | kubectl create -f -
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: user@example.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# ACME DNS-01 provider configurations
dns01:
# Here we define a list of DNS-01 providers that can solve DNS challenges
providers:
- name: cf-dns-prod
cloudflare:
email: user@example.com
# A secretKeyRef to a cloudflare api key
apiKeySecretRef:
name: cloudflare-api-key
key: api-key.txt
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: example-com-prod
namespace: cert-manager
spec:
secretName: example-com-tls-prod
issuerRef:
name: letsencrypt-prod
commonName: '*.example.com'
acme:
config:
- dns01:
provider: cf-dns-prod
domains:
- '*.example.com'
EOF
Happy secure web content serving for you :)
Wrap Up
In this blog post we learned how to install Heptio Contour Ingress Controller,
install Jetstack Cert-Manager and set it up to issue wildcard certs from Let's Encrypt, and with the help of AppsCode Kubed sync wildcard certs across labeled namespaces.
