The cluster that almost wasn't

Build, Break & Operate · Ship Boring Lab · Stage 1.

I set out to build a small Kubernetes cluster from scratch:

2 VMs → Ansible hardening → pinned Kubernetes v1.31 → `kubeadm init` + `join`

It sounded straightforward.

It wasn't.

What fought back

1. SSH — three rounds of debugging

The OrbStack VMs didn't have openssh-server installed.

Then I hit two more problems:

• The machines were running in isolated-machine mode
• The configured username was `spk`, not ubuntu

The breakthrough came through /mnt/mac for copying the required SSH key.

The fingerprints matched, but the server still refused the connection.

The sshd logs finally told me why.

Lesson: Before debugging SSH authentication, make sure there is actually an SSH server to authenticate against.

2. The clone trap

sb-node1 was created as a clone of sb-cp.

That meant it inherited:

• SSH host keys
• machine-id

Two machines should not share the same identity.

I regenerated the machine identity and SSH host keys.

With a new fingerprint, the connection worked.

Lesson: A cloned VM isn't automatically a new machine.

3. `swapoff -a` wasn't actually a failure

Ansible reported:

swapoff -a

There was no swap to disable. The machine was already in the desired state.

The automation needed to understand:

Already compliant = success.

That's the difference between a script that runs and automation that is actually idempotent.

4. Ansible exposed another assumption

This:

changed_when: 'stdout'

ran into problems with newer ansible-core.

The more portable pattern is to explicitly register the command result:

register: result

and evaluate the registered result.

Another small failure, another assumption removed.

5. One tab broke YAML

One tab inside a YAML heredoc was enough to kill the file.

Now this is part of the debugging reflex:

grep -Pn '\t'

Sometimes the problem isn't Kubernetes.

Sometimes it's one character.

The win

Eventually:

kubectl get nodes

returned the state I was after:

sb-cp       Ready
sb-node1    Ready

2/2 nodes Ready.

Kubernetes v1.31.4.

Running locally.

Zero dollars.

And every line of the setup is mine.

What I actually learned

Most of the learning came from debugging the assumptions underneath the Kubernetes installation:

• A VM isn't necessarily SSH-ready.
• A cloned machine isn't necessarily uniquely identified.
• A non-zero return code doesn't always mean the system is wrong.
• Automation needs to model desired state, not just command output.
• YAML doesn't care how small the mistake looks.

The automation was the easy part. The debugging is the skill.

That's what I want to capture with Ship Boring Lab: the failures, the assumptions, the fixes, and the reasoning that gets the system from "should work" to actually working.