Kubernetes: How to effectively backup etcd

Introduction

Build Docker Image

FROM alpine:latest

ARG ETCD_VERSION=v3.4.13

ENV ETCDCTL_ENDPOINTS "https://127.0.0.1:2379"
ENV ETCDCTL_CACERT "/etc/kubernetes/pki/etcd/ca.crt"
ENV ETCDCTL_KEY "/etc/kubernetes/pki/etcd/healthcheck-client.key"
ENV ETCDCTL_CERT "/etc/kubernetes/pki/etcd/healthcheck-client.crt"

RUN apk add --update --no-cache bash ca-certificates tzdata openssl

RUN wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz \
&& tar xzf etcd-${ETCD_VERSION}-linux-amd64.tar.gz \
&& mv etcd-${ETCD_VERSION}-linux-amd64/etcdctl /usr/local/bin/etcdctl \
&& rm -rf etcd-${ETCD_VERSION}-linux-amd64*

ENTRYPOINT ["/bin/bash"]
export ETCD_VERSION=v3.4.13
docker build --build-arg=$ETCD_VERSION -t etcd-backup:$ETCD_VERSION .
docker images | grep -i "etcd-backup"

Backup Etcd Datastore Using Kubernetes Cronjob

apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: etcd-backup
spec:
schedule: "*/5 * * * *"
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 5
concurrencyPolicy: Allow
jobTemplate:
spec:
template:
spec:
containers:
- name: etcd-backup
image: etcd-backup:v3.4.13
env:
- name: ETCDCTL_API
value: "3"
- name: ETCDCTL_ENDPOINTS
value: "https://127.0.0.1:2379"
- name: ETCDCTL_CACERT
value: "/etc/kubernetes/pki/etcd/ca.crt"
- name: ETCDCTL_CERT
value: "/etc/kubernetes/pki/etcd/healthcheck-client.crt"
- name: ETCDCTL_KEY
value: "/etc/kubernetes/pki/etcd/healthcheck-client.key"
command: ["/bin/bash","-c"]
args: ["etcdctl snapshot save /data/etcd-backup/etcd-snapshot-$(date +%Y-%m-%dT%H:%M).db"]
volumeMounts:
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
readOnly: true
- mountPath: /data/etcd-backup
name: etcd-backup
- mountPath: /etc/localtime
name: local-timezone
restartPolicy: OnFailure
hostNetwork: true
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
operator: Exists
- key: node.kubernetes.io/memory-pressure
effect: NoSchedule
operator: Exists
volumes:
- name: etcd-certs
hostPath:
path: /etc/kubernetes/pki/etcd
type: Directory
- name: etcd-backup
hostPath:
path: /data/etcd-backup
type: DirectoryOrCreate
- name: local-timezone
hostPath:
path: /usr/share/zoneinfo/Europe/Tallinn # mount host timezone to container
kubectl apply -f etcd-backup-cronjob.yaml
kubectl get cronjobs
NAME          SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
etcd-backup */5 * * * * False 0 <none> 4s
NAME                     COMPLETIONS   DURATION   AGE
etcd-backup-1614619800 1/1 2s 47s
ls -l /data/etcd-backup/
total 30224
-rw------- 1 root root 30945312 märts 1 19:30 etcd-snapshot-2021-03-01T19:30.db

Manage Cronjob Throught Gitlab CI/CD

Certified Kubernetes Administrator (CKA) & Linux System Administrator & CI/CD Integrations & Big Data: Apache Hadoop & Automating: Ansible, Bash, Python

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store