preloader

Quorum

Overview

Steps for dealing with etcd quorum issues with a TKG environment.

NOTE: This is by no means complete, I am just capturing some scrap notes after having to rescue a customer’s production TKGs control plane which had lost etcd quorum.

Steps

From one of the Control Plane nodes you are designating as the “leader” node:

  • Stop existing etcd container

  • Capture pod and container info

crictl inspect <pod-id> > pod.json
crictl inspect <container-id> > container.json
  • modify files to include --force-new-cluster
vim pod.json
vim container.json
  • Create the container, passing the ID of the previously-created pod, the container config file, and the pod config file. The ID of the container is returned.
crictl create $POD_ID container.json pod.json
  • Verify the container is running
crictl ps -a
  • Verify etcd container logs and test kube-apiserver
crictl logs $CONTAINER_ID

kubectl get nodes
  • Do some etcd stuff to clean-up
etcdctl member list
etcdctl member remove <member-id>
etcdctl endpoint health
  • Once healthy, take a snapshot
ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshot.db
  • Stop the container and drop the --force-new-cluster flag
crictl stop $CONTAINER_ID
vim container.json
  • Once healthy, fix-up the control-plane to be 1 node.
  • Then, once healthy, add additional etcd nodes back into cluster.
  • Go to the pub, it’s the customer’s shout for saving production.