$ mkdir -p certs
$ openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout certs/registry.key \
  -addext "subjectAltName = IP:10.211.55.250" \
  -x509 -days 3650 -out certs/registry.crt

The important bit here is that we set the subjectAltName to an IP number our MetalLB load balancer will assign. In our cluster we will assign 192.168.202.5 to the Docker Registry service (see later on).

Store Key and Certificate in Cluster

kubectl create secret tls registry-cert \
    --cert=certs/registry.crt \
    --key=certs/registry.key \
    -n distribution

The above command will store registry.crt and registry.key in a TLS secret called registry-cert in the namespace distribution. We will be deploying our Docker Registry in this namespace.

Create Persistant Volume for Docker Registry

---
# Namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: distribution

# StorageClass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: distribution-sc
provisioner: nfs.csi.k8s.io
parameters:
  server: 192.168.202.50
  share: /srv/distribution/
  # csi.storage.k8s.io/provisioner-secret is only needed for providing mountOptions in DeleteVolume
  # csi.storage.k8s.io/provisioner-secret-name: "mount-options"
  # csi.storage.k8s.io/provisioner-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4.1

---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: nfs.csi.k8s.io
  name: distribution-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: distribution-sc
  mountOptions:
    - nfsvers=4.1
  csi:
    driver: nfs.csi.k8s.io
    # volumeHandle format: {nfs-server-address}#{sub-dir-name}#{share-name}
    # make sure this value is unique for every share in the cluster
    volumeHandle: 192.168.202.50/srv/distribution##
    volumeAttributes:
      server: 192.168.202.50
      share: /srv/distribution

# PersistantVolumeClaim.yaml
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: distribution-pvc
  namespace: distribution
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  volumeName: distribution-pv
  storageClassName: distribution-sc

Create Namespace.yaml, StorageClass.yaml and PersistantVolumeClaim.yaml with the above content.

Deployment of registry container

# RegistryDeployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: registry
  name: registry
  namespace: distribution
spec:
  replicas: 1
  selector:
    matchLabels:
      run: registry
  template:
    metadata:
      labels:
        run: registry
    spec:
      nodeSelector:
        node-type: worker
      containers:
      - name: registry
        image: registry:2
        ports:
        - containerPort: 5000
        env:
        - name: REGISTRY_HTTP_TLS_CERTIFICATE
          value: "/certs/tls.crt"
        - name: REGISTRY_HTTP_TLS_KEY
          value: "/certs/tls.key"
        volumeMounts:
        - name: registry-certs
          mountPath: "/certs"
          readOnly: true
        - name: registry-data
          mountPath: /var/lib/registry
          subPath: registry
      volumes:
      - name: registry-certs
        secret:
          secretName: registry-cert
      - name: registry-data
        persistentVolumeClaim:
          claimName: distribution-pvc

Here we configure one pod to create a container from registry:2 image using the TLS key and certificate we created earlier. We tell the container that it can find those under the /certs directory. For this we use volume mounts. We define a read-only volumeMount with the name registry-certs to be mounted under /certs in the container. This registry-certs volumeMount references the secret volume registry-cert in the volumes section.

We also mount the persistent volume claim distribution-pvc under directory /var/lib/registry in the container.

Create Distribution Service

# Service.yaml
apiVersion: v1
kind: Service
metadata:
  name: distribution-service
  namespace: distribution
spec:
  type: LoadBalancer
  selector:
    # app.kubernetes.io/name: registry
    run: registry
  ports:
    - name: registry-tcp
      protocol: TCP
      port: 5000
      targetPort: 5000

Deployment

# kustomizastion.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: distribution-sc

resources:
  - Namespace.yaml
  - StorageClass.yaml
  - PersistantVolumeClaim.yaml
  - RegistryDeployment.yaml
  - Service.yaml

Create above files kustomization.yaml and start the deployment:

$ kubectl create -k .

Check deployment status with

$ kubectl -n test get all

Additional things to do

Since we are dealing with a self signed certificate, we need to instruct the host operating system to trust this certificate. Normally you would do that with a Certificate Authority (CA) certificate, but since this is self signed, the certificate is both the certificate for the registry service, as well as its own CA. So let’s load the registry.cert as a new CA on all the cluster nodes. We need to do this because the k3s service on all the cluster nodes will need to be able to pull from this repository, and it must trust this certificate. We can easily do this using Ansible:

ansible -i hosts nodes -b -m copy -a "src=certs/registry.crt dest=/usr/local/share/ca-certificates/registry.crt"
ansible -i hosts nodes -b -m shell -a "update-ca-certificates"

Final step is do restart the kubernetes-service on all nodes to make them use the self-signed certificates.

Testing

$ docker pull alpine
$ docker tag alpine 10.211.55.250:5000/alpine:latest
$ docker push 10.211.55.250:5000/alpine:latest

With this, you pull acontainer image from the official docker-hub and push it into our newly generated local distribution registry.

Let’s now try to deploy a pod with this new image. Create a file called alpine-test-pod.yaml with the following content:

alpine-test-deployment.yaml
apiVersion: v1
kind: Pod
metadata:
  name: alpine-test
#  namespace: test
spec:
  containers:
  - name: alpine
    image: 192.168.137.250:5000/alpine:latest
    command: ["sleep", "60s"]
  restartPolicy: "Never"

Deploy it:

$ kubectl create -f alpine-test-deployment.yaml

Check the state of the pod:

$ kubectl describe pod alpine-test

README.md Unescape Escape

Deploying Docker Registry on Kubernetes

Howto

Create Key and Certificate