Reverse Proxy¶

Since we only have a single public IP, if we want to set up multiple public services we will need a reverse proxy to redirect the requests to the correct services. Our proxy will also handle the ssl termination, to simplify the set up of the services.

For managing our certificates we will be using cert-manager and as a proxy we will be using Traefik.

Note

Traefik is the default proxy from k3s, but to have more granular control we are disabling it and we are going to deploy it as Helm chart

Traefik¶

Similarly, to deploy Traefik we have to execute:

kubectl create namespace proxy
helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm install -n proxy traefik traefik/traefik -f traefik-values.yaml --version 26.1.0

Note

In the logs we might have a warning and we might have to increase the rmeem_max ansible all -b -m shell -a "sysctl -w net.core.rmem_max=2500000". To retrieve the logs do kubectl -n proxy logs -f -l "app.kubernetes.io/name=traefik"

# https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml

image:
  #tag: "v2.11.0"
  pullPolicy: IfNotPresent

deployment:
  enabled: true
  kind: Deployment
  replicas: 2

ingressClass:
  enabled: true
  isDefaultClass: true

# We will create our IngressRoute for the dashboard
ingressRoute:
  dashboard:
    enabled: False

providers:
  kubernetesCRD:
    enabled: true
    allowCrossNamespace: true
    allowExternalNameServices: true
  kubernetesIngress:
    enabled: true


additionalArguments:
  # Used for the generation of the certificates
  - "--providers.kubernetesIngress.ingressClass=traefik-cert-manager"
  - "--providers.kubernetesingress.allowexternalnameservices=true"

logs:
  general:
    level: INFO
    format: json
  access:
    enabled: true
    format: json

metrics:
  prometheus:
    entryPoint: metrics
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    manualRouting: false
    service:
      enabled: false


# TLS Options are created as TLSOption CRDs
# https://doc.traefik.io/traefik/https/tls/#tls-options
# Example:
tlsOptions:
  default:
    sniStrict: true
    preferServerCipherSuites: true
    cipherSuites:
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
    curvePreferences:
      - CurveP521
      - CurveP384

ports:
  websecure:
    http3:
      enabled: true
  metrics:
    # It has to be the same with the traefik-dashboard-service.yaml in monitoring
    port: 9100

service:
  enabled: true
  type: LoadBalancer
  spec: 
    externalTrafficPolicy: Local
    loadBalancerIP: "192.168.179.240"

resources:
  requests:
    cpu: "250m"
    memory: "100Mi"

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 5

Now that our proxy is running we can define some Middlewares to simplify the deployment of services:

kubectl apply -f https_redirect.yaml
kubectl apply -f security_headers.yaml

Traefik will be my main proxy that will forward requests to other services. So we also deploy these requirements:

kubectl apply -f grigoris_proxy.yaml
kubectl apply -f box_proxy.yaml
kubectl apply -f traefik-dashboard.yaml

Cert-manager¶

Since we want high availability, we need also to handle our certificates in a way that can be hanlded from traefik. Traefik offers this support in the enteprise version, but since I am cheap, I will use an another manager (i.e. cert-manager)

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager -f cert-values.yaml --version v1.14.3
kubectl apply -f cert-vpa.yaml

With cert-values.yaml:

#https://github.com/jetstack/cert-manager/blob/master/deploy/charts/cert-manager/values.yaml

installCRDs: true
replicaCount: 1

#https://github.com/jetstack/cert-manager/issues/959
podDnsPolicy: "Default"

prometheus:
  enabled: true
  servicemonitor:
    enabled: true

Note

Initially the prometheus monitoring should not be activate until we deploy the monitoring stack. Afterwards we can activate it.

Now that the cert-manager is running we can create our certificate issuers:

kubectl apply -f self_signed.yaml
kubectl apply -f letenrcypt_stagging.yaml
kubectl apply -f letenrcypt.yaml

Note

We are using traefik-cert-manager as name for the traefik ingress. That we will have to specify it when we deploy the traefik

Let`s encrypt with wildcards¶

Cert manager has out of the box support for the basic HHTP01 challenge. This does not allow to create certificates with wildcards. For that we have to use the DNS01 challenge. But we need an extra plugin for that purpose.

# THESE INSTRUCTIONS DO NOT WORK. THEY ARE NOT READY
helm repo add bwolf https://bwolf.github.io/cert-manager-webhook-gandi
helm install webhook-gandi bwolf/cert-manager-webhook-gandi --namespace cert-manager --version v0.2.0 --set features.apiPriorityAndFairness=true --set logLevel=2

kubectl -n general annotate secret gandi kubed.appscode.com/sync="app=gandi-key" --overwrite
kubectl label namespace cert-manager app=gandi-key

Testing that everything works¶

To test that everything works, we can deploy a testing whoami service:

kubectl apply -f whoami.yaml
# Check that everything works and the delete the service
kubectl delete -f whoami.yaml
# Forcing renew. Good practice when updating the cert-manager to make sure that
# everything still works
kubectl cert-manager status certificate login.anagno.me -n authentication
kubectl cert-manager renew login.anagno.me -n authentication

Resources:

https://medium.com/dev-genius/setup-traefik-v2-for-ha-on-kubernetes-20311204fa6f
https://traefik.io/blog/install-and-configure-traefik-with-helm/
https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
https://www.thebookofjoel.com/k3s-cert-manager-letsencrypt
https://cert-manager.io/docs/usage/kubectl-plugin/