Quick post to help anyone else stuck in the same situation I was today. The internet, or at least my Googlefu, failed me.
Today we had a Kubernetes deployment get stuck, all health checks seemed healthy, but a deployment got stuck somewhere between demoting the current replica set to be the old one, and creating a new one.
This seemed to put all replica set creation in limbo as other deployments also ended up stuck in a similar state.
All health checks I could think of came back clear. Attempts to undo the rollout
seemed to be partially actioned with the command
kubectl rollout status deployment/app returning the message:
Waiting for deployment spec update to be observed...
Nothing I tried seemed to help, eventually I resorted to
kill-ing one of the
controller-manager processes on a master node which seemed to kick things back
to life. This seemed an extreme measure but I’d run out of non-destructive ideas
by this point.
My wild speculation is that an in-memory lock was blocking the replica set creation and that killing the process broke the lock.
Hopefully this helps someone else, or me, in the future.