Garry Shutler

G

Kubernetes rollout stuck

November 14, 2018 · 1 min read

Quick post to help anyone else stuck in the same situation I was today. The internet, or at least my Googlefu, failed me.

Today we had a Kubernetes deployment get stuck, all health checks seemed healthy, but a deployment got stuck somewhere between demoting the current replica set to be the old one, and creating a new one.

This seemed to put all replica set creation in limbo as other deployments also ended up stuck in a similar state.

All health checks I could think of came back clear. Attempts to undo the rollout seemed to be partially actioned with the command kubectl rollout status deployment/app returning the message:

Waiting for deployment spec update to be observed...

Nothing I tried seemed to help, eventually I resorted to kill-ing one of the controller-manager processes on a master node which seemed to kick things back to life. This seemed an extreme measure but I’d run out of non-destructive ideas by this point.

My wild speculation is that an in-memory lock was blocking the replica set creation and that killing the process broke the lock.

Hopefully this helps someone else, or me, in the future.

hero image

Hey, I’m Garry Shutler

CTO and co-founder of Cronofy.

Husband, father, and cyclist. Proponent of the Oxford comma.

As well as longer form thoughts here, I post shorter thoughts, puns, and bad jokes on Twitter.


Cronofy is hiring! Check out our Engineering Roles.

email icon twitter icon github icon