Revision Viewer for etcd Snapshots and Kubernetes Objects

Open an etcd snapshot, or a member’s own database file, and see what fills it: how close it is to the quota, which keys and Kubernetes resources take the space, how much is old revisions that compaction would remove, how much is free pages that a defrag would give back, Secrets stored in the clear, leases, members and alarms. Pick any key to see every revision the file keeps and what changed in each. Kubernetes objects show the way kubectl get -o yaml shows them. It all runs in your browser.

Nothing you open leaves this page. The tool doesn’t send anything anywhere, and the page’s security policy stops it from fetching or loading anything from another site. A snapshot holds every Secret a cluster has that isn’t encrypted at rest, so it stays on your machine.

On a wide screen you can open it full screen.

How to Use It

  1. Get a snapshot. On a machine that can reach etcd, etcdctl snapshot save writes one. On a kubeadm control plane node:
ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key snapshot save backup.db
  1. Open it. Drop the file on the page or choose it. A member’s own member/snap/db file works too.
  2. Set the quota if your cluster runs with a --quota-backend-bytes other than the 2 GiB default. Pick it from the list, or pick Another size and type it, such as 6GiB.
  3. Read what to look at, then the space by resource, kind and prefix. Pick a resource or a prefix to list its keys.
  4. Pick a key to see its revisions. Pick a revision to see its value, or what changed from the one before.

Keep a snapshot as safe as the cluster’s credentials: it holds every Secret that isn’t encrypted at rest.

What etcd Keeps

etcd is the database behind Kubernetes. Every object in a cluster, from each Pod to each Secret, is a key in etcd under /registry/, and kube-apiserver is the only thing that reads and writes it.

etcd never overwrites a value. Each write adds a new revision of the key and leaves the old one where it was, so a client can read the past or watch every change. Old revisions stay until compaction removes them. kube-apiserver compacts every five minutes, but a busy key can pile up many revisions in that time: a node’s heartbeat lease changes every ten seconds.

Compaction frees space inside the file without making the file smaller. The space turns into free pages that etcd reuses for new writes, and only defragmenting a member gives them back to the disk. All of it counts toward the quota. When the database reaches it, etcd raises the NOSPACE alarm and takes only reads and deletes, and for a Kubernetes cluster that means nothing can change: no new Pods, no updated leases, no events.

The data lives in one file per member, in a format called bbolt: fixed-size pages that hold B+trees. A snapshot is a copy of that file with a SHA-256 hash at the end. Kubernetes objects inside it are stored as protobuf, which no person can read, so the viewer decodes them with Kubernetes’ own field names and prints them as kubectl does. Two revisions of an object then differ by the lines that changed.

How It Was Tested

The viewer was checked against real etcd servers, etcd’s own tools and Kubernetes’ own code. A script ran etcd 3.6.15, 3.5.34 and 3.4.45 and filled each the way a Kubernetes cluster does: Pods, Nodes, Deployments, Services, leases renewed every ten seconds, events, Secrets in the clear and encrypted at rest, and custom resources, with the objects encoded by Kubernetes 1.37.1’s own packages, exactly as kube-apiserver stores them. Then it compacted, wrote more history, deleted keys and saved snapshots.

  • etcd’s figures. For each snapshot the viewer gives the same hash, size, revision and key count as etcdutl snapshot status, and the same free pages and page counts as bbolt’s own command line.
  • Keys and revisions. Every live key matches what etcdctl get returns: value, create and mod revision, version and lease. Every older revision etcd could still serve for three busy keys is in the history, with the same value.
  • Kubernetes objects. All 141 objects in the snapshot, of 27 kinds, decode to the same JSON that Kubernetes’ Go packages make of the same bytes, and 139 print as YAML byte for byte as kubectl prints them. kubectl itself can’t print the other two. 240 more objects full of awkward strings print exactly as kubectl prints them.
  • Special cases. A database that filled its quota and raised NOSPACE, a member’s file with no hash at the end, authentication switched on, a changed byte and a cut file.

A 198 MiB snapshot with 250,000 revisions opens in about two seconds on the command line and three in the browser.

The Code

The viewer’s logic is one JavaScript file with no dependencies, open source under the Apache License 2.0. It also runs as a command line in Node.js, where a backup job can check each snapshot it saves:

node revisions/cli.js backup.db --quota 8GiB
node revisions/cli.js backup.db --history /registry/leases/kube-node-lease/node-1
node revisions/cli.js backup.db --keys > keys.csv

The full manual, the tests and the snapshots they replay are in the revisions folder on GitHub. The other tools work the same way.