Kubernetes rollout stuck
November 14, 2018 · 1 min read
Quick post to help anyone else stuck in the same situation I was today. The internet, or at least my Googlefu, failed me.
Today we had a Kubernetes deployment get stuck, all health checks seemed healthy, but a deployment got stuck somewhere between demoting the current replica set to be the old one, and creating a new one.
This seemed to put all replica set creation in limbo as other deployments also ended up stuck in a similar state.
All health checks I could think of came back clear. Attempts to undo the rollout
seemed to be partially actioned with the command
kubectl rollout status deployment/app returning the message:
Waiting for deployment spec update to be observed...
Nothing I tried seemed to help, eventually I resorted to
kill-ing one of the
controller-manager processes on a master node which seemed to kick things back
to life. This seemed an extreme measure but I’d run out of non-destructive ideas
by this point.
My wild speculation is that an in-memory lock was blocking the replica set creation and that killing the process broke the lock.
Hopefully this helps someone else, or me, in the future.
Hey, I’m Garry Shutler
CTO and co-founder of Cronofy.
Husband, father, and cyclist. Proponent of the Oxford comma.
As well as longer form thoughts here, I post shorter thoughts, puns, and bad jokes on Mastodon and Twitter.
Cronofy is hiring!
Find out about Cronofy and our open roles.