Kichul Kang

Building a Bare-Metal Kubernetes Cluster on Raspberry Pis, Part 3

#Kubernetes#Infrastructure

In Part 2, the three Pis became an HA control plane. Next on the list was Argo CD, which will start changing the cluster on its own. Before giving anything that kind of access, I wanted a way back.

Backups before anything else

The only backup I had was the one-off snapshot from before the joins, by then 11 days old. Three etcd members don’t change that. Replication protects against a dead node, not a bad change: if I delete a namespace by mistake, the deletion reaches all three members in milliseconds. A snapshot from before the change is the only way back.

Managed Kubernetes backs up etcd for you, and k3s takes a snapshot every 12 hours by default. kubeadm gives you etcdctl snapshot save and leaves the schedule to you.

The plan has two layers, because each covers a failure the other can’t:

On the cluster On my Mac
Protects against A bad change or a broken upgrade Losing the cluster itself: SD cards, a Pi, power
Runs Every 6 hours, always Every 6 hours, when the Mac is awake and at home
Keeps Newest 8 per node, about 2 days Newest 28, about 7 days

A snapshot is small. etcd’s database was 6.8MB, so four a day add almost nothing to the roughly 12GiB a day etcd already writes to each card.

A CronJob on the cluster

The job runs every 6 hours on whichever control plane node Kubernetes picks. It uses the host network, so 127.0.0.1:2379 is that node’s own etcd member, and any of the three has the full data.

The etcd image already on every node has etcdctl, but no shell, so it can’t name a file or delete an old one. The job is split in two: an init container takes the snapshot, and a small busybox container saves it with a timestamp, saves a copy of the node’s certificates next to it, and prunes old ones:

initContainers:
  - name: snapshot
    image: registry.k8s.io/etcd:3.6.8-0
    command:
      - etcdctl
      - --endpoints=https://127.0.0.1:2379
      - --cacert=/etc/kubernetes/pki/etcd/ca.crt
      - --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt
      - --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
      - snapshot
      - save
      - /work/snapshot.db
containers:
  - name: store
    image: docker.io/library/busybox:1.37.0
    command:
      - sh
      - -euc
      - |
        umask 077
        ts=$(date -u +%Y%m%dT%H%M%SZ)
        mv /work/snapshot.db "/backups/etcd-$NODE_NAME-$ts.db"
        tar czf "/backups/pki-$NODE_NAME-$ts.tar.gz" -C /etc/kubernetes pki
        for prefix in etcd pki; do
          ls -1t /backups/$prefix-* | tail -n +9 | xargs -r rm -f --
        done
        ls -lh /backups

/work is an empty directory the two containers share, and /backups is /var/backups/etcd on the node. The rest of the manifest is safety settings: concurrencyPolicy: Forbid so two backups never overlap, a 5-minute deadline, two retries, and memory limits so a backup can’t starve the etcd it’s copying. I ran it once by hand to check it:

# On my Mac
$ kubectl -n kube-system create job etcd-backup-manual-1 --from=cronjob/etcd-backup
$ kubectl -n kube-system logs job/etcd-backup-manual-1 --all-containers
...
Snapshot saved at /work/snapshot.db
-rw-------    1 root     root        6.5M Oct 24 08:26 etcd-node-3-20261024T082647Z.db
-rw-------    1 root     root       20.1K Oct 24 08:26 pki-node-3-20261024T082647Z.tar.gz

The files are readable only by root, because a snapshot holds every Secret in the cluster.

Why the certificates too

A snapshot alone can’t rebuild the cluster. Every node, kubeconfig and service account token trusts the certificate authorities in /etc/kubernetes/pki: ca.key for the cluster, etcd/ca.key for etcd, sa.key for service account tokens, and front-proxy-ca.key for requests the API server forwards to add-ons like metrics-server. Rebuilt with new CAs, the restored data would point at an identity system that no longer exists. With the old keys, it’s the same cluster.

A copy on my Mac

The CronJob’s files are still on the SD cards, so they die with them. The real backup has to live somewhere else. kubeadm’s etcd listens on each node’s own address as well as 127.0.0.1, so my Mac can take a snapshot directly over the network, with a copy of the etcd CA and the healthcheck client certificate. I used etcd’s official 3.6.8 release for macOS, checksum-verified, because Homebrew only had 3.7, and a restore is a bad time to discover a version mismatch.

The script tries node-1, then node-2, then node-3, so one dead node doesn’t stop it. It saves to a temporary file and only keeps the snapshot if etcdutl snapshot status can read it:

for ip in "${NODES[@]}"; do
  if etcdctl --endpoints="https://$ip:2379" --cacert=... --cert=... --key=... \
      --dial-timeout=5s --command-timeout=120s snapshot save "$tmp"; then
    if etcdutl snapshot status "$tmp" -w table; then
      mv "$tmp" "$DEST/etcd-$ts.db"
      # keep the newest 28, then stop
      exit 0
    fi
  fi
  rm -f -- "$tmp"
done

launchd, macOS’s scheduler, runs it at login and every 6 hours. If the Mac was asleep, missed runs are made up once on wake. Its log shows a laptop’s schedule, not a server’s:

2026-10-24 01:28:00 saved etcd-20261024T082754Z.db from 10.0.0.150 (7.1M)
2026-10-24 08:36:18 saved etcd-20261024T153617Z.db from 10.0.0.150 (6.5M)
2026-10-24 17:52:07 saved etcd-20261025T005206Z.db from 10.0.0.150 (6.5M)
2026-10-25 17:08:18 saved etcd-20261026T000817Z.db from 10.0.0.150 (6.5M)

That’s the tradeoff of keeping the off-cluster copy on a laptop: it’s only as fresh as the last time I was home. The cluster copies fill the gaps. I also saved one copy of node-1’s /etc/kubernetes/pki to the Mac. The CAs are shared by all three nodes and last until 2036, so one copy is enough.

Everything in that folder is as sensitive as a root password: every Secret, and the keys to mint new cluster identities. It stays out of synced folders, on an encrypted disk.

Testing a restore

A backup that has never been restored is a guess. A real restore of a three-member etcd means stopping every API server, so I tested the file instead: restored the newest snapshot into a throwaway single-member etcd on my Mac, started it on unused ports, and looked inside:

# On my Mac
$ etcdutl snapshot restore etcd-20261024T082754Z.db --data-dir /tmp/restore/data --name restore-test \
    --initial-cluster restore-test=http://127.0.0.1:23800 --initial-advertise-peer-urls http://127.0.0.1:23800
$ etcd --name restore-test --data-dir /tmp/restore/data --listen-client-urls http://127.0.0.1:23790 ... &
$ etcdctl --endpoints=http://127.0.0.1:23790 get / --prefix --keys-only | grep -c .
414

It had all 414 keys: the five namespaces, the three nodes, the Cilium operator, CoreDNS and metrics-server deployments, and the etcd-backup CronJob itself, which had existed for about a minute when the snapshot was taken. The file was complete and current. Restoring the live cluster from it is a drill for another day.

Argo CD

In Part 1, I installed Argo CD only to measure it. This time it runs the cluster. Until now, everything on the cluster got there through a kubectl apply I typed, and the only full record of what was running was the cluster itself. With Argo CD, a Git repo says what should exist, and Argo CD keeps the cluster matching it: it renders the repo, compares the result with what the API server has, and applies the difference.

The repo

The repo is public at github.com/kckang1103/homelab, so Argo CD can read it without credentials. That also means nothing secret can ever go in it. Snapshots, keys and kubeconfigs stay in the folder on my Mac.

homelab/
├── bootstrap/root.yaml      applied by hand, once
├── apps/                    one Application per component
│   ├── argocd.yaml
│   └── etcd-backup.yaml
├── argocd/                  Argo CD itself
│   ├── kustomization.yaml
│   └── namespace.yaml
└── etcd-backup/
    └── cronjob.yaml

An Application is a custom resource that Argo CD adds to Kubernetes. It says where the manifests live and where to deploy them:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: etcd-backup
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/kckang1103/homelab.git
    targetRevision: main
    path: etcd-backup
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    automated: { prune: true, selfHeal: true }

automated makes Argo CD sync on its own instead of waiting for someone to press Sync, prune deletes objects that were removed from Git, and selfHeal reverts changes made by hand.

Argo CD only looks at folders some Application points at, so something has to point at apps/. That’s root, an Application whose path is apps/. The files in that folder are themselves Applications, so this layout is called app of apps:

root (applied by hand)                  watches apps/
├── argocd                              watches argocd/  → Argo CD's own install
└── etcd-backup                         watches etcd-backup/  → the CronJob

Adding a component later means adding a file to apps/ and a folder next to it, then pushing. root itself lives in bootstrap/, which no Application watches. It’s the one object that has to be created by something other than Argo CD.

Installing Argo CD from the repo

argocd/kustomization.yaml doesn’t copy Argo CD’s manifests. It points at the upstream install.yaml and lists the changes I want on top. Kustomize is built into kubectl, and its output is plain YAML:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
  - namespace.yaml
  - https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.6/manifests/install.yaml
patches:
  - target: { kind: Deployment, name: argocd-dex-server }
    patch: |
      - op: add
        path: /spec/replicas
        value: 0
  - target: { kind: Deployment, name: argocd-notifications-controller }
    patch: |
      - op: add
        path: /spec/replicas
        value: 0

In Part 1 I installed from stable, which changes whenever a new release ships. Here the URL names a release, v3.5.6, so the same commit always installs the same version, and an upgrade is a one-line change in Git. The two patches do what kubectl scale did in Part 1, turning off Dex (single sign-on) and notifications, but in a form that survives a sync. They use add rather than replace because upstream doesn’t set replicas at all, and replace fails on a field that doesn’t exist.

The namespace: line matters more than it looks. Upstream’s manifests leave the namespace off so you can choose one. Rendered with the line, 59 objects say namespace: argocd; without it, only 3 do, all inside ClusterRoleBindings. A plain kubectl apply would then put the Deployments in default while the permissions point at argocd. It would install without errors and then not work.

Bootstrapping

Argo CD can’t install itself before it exists, so the first install is plain kubectl, from the same folder:

# On my Mac
$ kubectl apply -k argocd/ --server-side
$ kubectl apply -f bootstrap/root.yaml

The first command creates the namespace, the CRDs and about 60 other objects. Kubernetes starts the pods like any other Deployment, and Argo CD comes up with nothing to manage. The second command gives it root, and from there the creation timestamps show the chain:

Object Created (UTC)
Namespace argocd and the CRDs 01:17:50
Argo CD’s Deployments 01:17:54
Application root, applied by hand 01:23:02
Applications argocd and etcd-backup, created by root 01:23:05

In those 3 seconds, Argo CD fetched the repo, rendered apps/, found two Applications missing and created them. Each of those then found its objects already on the cluster with matching content, from the hand-applied install and the CronJob I’d created earlier. Adopting them only meant adding Argo CD’s tracking annotation. Nothing restarted.

From then on, Argo CD manages its own install. That’s the one app where a bad commit could take down the tool I’d use to fix it, so it has two differences from the others:

  syncPolicy:
    automated: { prune: false, selfHeal: true }
    syncOptions:
      - ServerSideApply=true

With pruning off, a commit that made argocd/ render nothing couldn’t delete the argocd namespace with Argo CD inside it. Server-side apply is the same fix as --server-side above: a normal apply stores a copy of each object in an annotation, and some of Argo CD’s CRDs are larger than the 256KB annotation limit.

What it costs

# On my Mac
$ kubectl -n argocd top pods
NAME                                                CPU(cores)   MEMORY(bytes)
argocd-application-controller-0                     3m           245Mi
argocd-applicationset-controller-58bb5c7479-ht2tp   1m           22Mi
argocd-redis-869b85489d-vf9bv                       8m           8Mi
argocd-repo-server-5bf66565cd-zh9dr                 1m           78Mi
argocd-server-7474bb8f76-ggwq7                      2m           27Mi

That’s 380Mi in total, close to Part 1’s estimate from summing resident sizes. The application controller is most of it: it keeps a copy of every object it manages in memory, so it can compare them with Git without asking the API server each time. Argo CD runs once for the whole cluster, so the cost isn’t multiplied by three.

node-2 rebooted during the install

While the install was rolling out, kubectl printed watch ended with error twice, and argocd-server restarted once. Its last log line before the restart:

{"level":"fatal","msg":"Get \"https://10.96.0.1:443/apis/argoproj.io/v1alpha1/namespaces/argocd/appprojects/default\": dial tcp 10.96.0.1:443: i/o timeout","time":"2026-10-26T01:20:07Z"}

10.96.0.1 is the kubernetes Service, which forwards to the three API servers. Every pod on node-2 showed exactly one restart, and the events said why:

Warning   Rebooted   node/node-2   Node node-2 has been rebooted, boot id: 25db0f71-1ccf-4409-a4d2-ece10b86ab1a

The kubelet compares the kernel’s boot ID with the one it saw last, so this was a real reboot, not just a kubelet restart. The cluster recovered without help: all three nodes went back to Ready, and all three etcd members reported healthy. etcd kept a quorum with two of three members, which is the HA from Part 2 doing its job.

Finding the cause didn’t go as well. I couldn’t SSH into the node, so I used kubectl debug node/node-2 --profile=sysadmin, which starts a pod on the node with the host’s filesystem at /host. Running chroot /host journalctl then reads the node’s own system log. It came back with:

Specifying boot ID or boot offset has no effect, no persistent journal was found.

Raspberry Pi OS keeps the journal in memory, which is gentler on the SD card, so the reboot erased the logs from before it. What was left:

  • systemd runs the Pi’s hardware watchdog with a 1-minute timeout. If the system stops responding for a minute, for example stuck waiting on the SD card, the chip resets the board.
  • vcgencmd get_throttled returned 0x0, so no undervoltage since boot. The counter resets at boot, though, so it says nothing about the moment of the reboot.

The timestamps had their own trap. The current boot’s first journal line was stamped 01:15:29, two minutes before the install started. But a few minutes later the node reported only 2 minutes of uptime, which puts the real boot at about 01:20. The Pi 5’s clock has no battery: the kernel log shows it starting at 1970, and systemd then sets the clock to the last time it saved, until NTP corrects it. Every timestamp from early in a boot can be minutes behind. With that corrected, the reboot came during the install, while the nodes were pulling Argo CD’s images. That’s the same kind of SD card load that caused trouble in Parts 1 and 2.

That isn’t the whole story, though. When I checked again six days later, node-2 had a different boot ID. It rebooted a second time at 02:45 UTC the same night, about 85 minutes after the first, and nothing heavy was running then. node-1 and node-3 have the same boot IDs as before. Two resets on one node, one of them without any load, points at that node’s power supply or SD card more than at the install. It’s been stable since. I can’t say more without logs, so the fix for now is making sure the next one leaves evidence, by turning on a persistent journal on each node:

The usual advice is to create /var/log/journal, because journald’s default, Storage=auto, writes to disk only if that directory exists. I did that on node-1, restarted journald, and the journal stayed in /run. journald reads its settings from several files, and systemd-analyze cat-config prints the combined result with each file’s path:

# On node-1
$ systemd-analyze cat-config systemd/journald.conf | grep -iE '^# /|Storage'
# /etc/systemd/journald.conf
#Storage=auto
# /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf
Storage=volatile
...

Raspberry Pi OS ships a config file that forces the journal into memory, whether the directory exists or not. Files in journald.conf.d are applied in name order, with later ones overriding earlier ones, so a file named 90-… wins:

# On each node
$ sudo mkdir -p /etc/systemd/journald.conf.d
$ printf '[Journal]\nStorage=persistent\nSystemMaxUse=200M\n' | sudo tee /etc/systemd/journald.conf.d/90-persistent.conf
$ sudo systemctl restart systemd-journald && sudo journalctl --flush
$ journalctl --header | grep -i 'file path' | head -1
File path: /var/log/journal/2ab4625dcdce4a3c99dd34a8d1fb15f4/system.journal

SystemMaxUse=200M caps it. By default, journald would take up to 10% of the card, about 1.5GB, which is a lot next to the 6 to 8GB free on each node.

Deploying an app with git push

To see the whole loop work, I deployed a small nginx app without running kubectl apply. It took two files: apps/hello.yaml, an Application pointing at hello/ with CreateNamespace=true so Argo CD creates the namespace, and hello/deployment.yaml, a plain Deployment with no namespace: line, since the Application’s destination supplies it. Then a commit and a push.

Nothing happened right away. A few minutes later, the apps didn’t agree on which commit was current:

NAME          SYNC     HEALTH    REV                                        RECONCILED
argocd        Synced   Healthy   8a5c0d004cde99537ca1791142d9b32119044729   2026-11-01T07:46:37Z
etcd-backup   Synced   Healthy   47f18836c73b4f47e712924e01c744590b86586c   2026-11-01T07:47:53Z
root          Synced   Healthy   8a5c0d004cde99537ca1791142d9b32119044729   2026-11-01T07:46:18Z

etcd-backup had already seen my commit, 47f1883, but its folder hadn’t changed, so it had nothing to do. root, the app that had to notice the new file, hadn’t checked Git since before the push. Argo CD doesn’t get notified of a push. It polls, and each app has its own timer: every 120 seconds plus a random delay of up to 60, so they don’t all fetch at once. A webhook from GitHub would make it instant, but that needs Argo CD reachable from the internet, which mine isn’t. To skip the wait, I annotated root:

# On my Mac
$ kubectl -n argocd annotate application root argocd.argoproj.io/refresh=normal --overwrite

Seconds later, hello appeared, synced, and its pod was running on node-2. root had created the hello Application, and hello had created the Deployment, the same two steps as the bootstrap.

Self-heal

Then I changed it the wrong way, with kubectl -n hello scale deployment hello --replicas=3. I expected the extra pods to last until the next poll. They were terminating almost immediately.

Argo CD watches the cluster continuously, not on a timer. The application controller keeps a watch open on every object it manages, so the scale was reported within a second. It compared that with the manifests it already had cached for the current commit, found 3 where Git said 1, and re-applied Git after a 2-second backoff. The backoff grows if the same change keeps coming back, up to 5 minutes, so Argo CD doesn’t fight another tool in a tight loop. So there are two loops: changes in Git are picked up every 2 to 3 minutes, and changes on the cluster are reverted within seconds.

The right way to scale is the same as the right way to deploy. I changed replicas to 2 in hello/deployment.yaml and pushed, and this time the hello app picked it up, since the change was in its folder, and the second pod stayed.

Removing an app

None of my Applications had a finalizer, so deleting apps/hello.yaml would have made root delete the hello Application and nothing else. The Deployment would keep running, with nothing managing it anymore. That default is deliberate: one bad commit to apps/ shouldn’t be able to wipe out a running service.

To remove the app completely, I first added Argo CD’s finalizer to the Application, in its own commit:

metadata:
  name: hello
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io

A finalizer tells the API server that a controller has cleanup to do before an object can be deleted. When root then pruned hello, the API server only marked it for deletion. Argo CD deleted the Deployment, Kubernetes deleted its pods, then Argo CD removed its finalizer and the Application went away. The app’s last events show the cleanup, all within about a second:

Updated health status: Healthy -> Progressing
Updated sync status: Synced -> OutOfSync
Updated health status: Progressing -> Missing

The finalizer had to be its own commit. If I’d added it and deleted the file in the same commit, the live Application would never have had it. The hello namespace stayed behind, because CreateNamespace=true creates it without tracking it, so that one I deleted by hand.

What’s next

The cluster now has a way back and a record of what should be running on it. What it doesn’t have is a view of what’s happening. node-2’s reboots were only noticed because I happened to be watching. Next:

  • Monitoring: Prometheus and Grafana, deployed through Argo CD from a Helm chart, and sized for 4GB nodes and SD cards: short retention, slower scrapes and memory limits.
  • node-2: with the journal now on disk, the next reboot should explain itself. Until then, I’ll swap its power supply with another node’s to see whether the problem follows the supply.
  • A real restore drill: restoring the live three-member etcd from a snapshot, rather than just checking the file.

← All posts