Robust Software

Tales of a code samurai

Kubernetes Rollout Stuck

Quick post to help anyone else stuck in the same situation I was today. The internet, or at least my Googlefu, failed me.

Today we had a Kubernetes deployment get stuck, all health checks seemed healthy, but a deployment got stuck somewhere between demoting the current replica set to be the old one, and creating a new one.

This seemed to put all replica set creation in limbo as other deployments also ended up stuck in a similar state.

All health checks I could think of came back clear. Attempts to undo the rollout seemed to be partially actioned with the command kubectl rollout status deployment/app returning the message:

Waiting for deployment spec update to be observed...

Nothing I tried seemed to help, eventually I resorted to kill-ing one of the controller-manager processes on a master node which seemed to kick things back to life. This seemed an extreme measure but I’d run out of non-destructive ideas by this point.

My wild speculation is that an in-memory lock was blocking the replica set creation and that killing the process broke the lock.

Hopefully this helps someone else, or me, in the future.

Hiring for the Future

When hiring people, beyond the basic competency requirement of the role, I look for an alignment in future goals. This is why the question of “where do you see yourself in N years time?” is so ubiquitous.

Ruby 2 - Module#prepend

As you may know Ruby 2.0.0 has been released. Despite the major version it is mostly an incremental release. However, it does include a few breaking changes so a major version is warranted. However, whilst the usefulness of most of the new features was obvious to me, I couldn’t say the same of Module#prepend.

However, whilst listening to the Ruby Rogues podcast on Ruby 2 one of the Rogues (I think it was Josh Susser) referred to memoization as a case where it would make a difference. Because of that I thought I would implement memoization in 1.9.3 and then in 2.0.0 using Module#prepend and see what came out of it. The process led to a personal “ah-ha” moment and so I thought I’d share what I found.


If there was one thing that I learnt in 2012 that I would want to convey to all the developers I know, it would be this:

Logging is about so much more than failures

I don’t know if it was just my experience but little to no emphasis was put on logging aside from handling exceptions. In the .NET world this boiled down to adding ELMAH to your project and then forgetting about it, with Rails it meant having nothing but the logs that you got out of the box. I’ve found that if that’s all you have, you’re really missing out.

Minimalism in an Age of Tremendous Hardware

Usually – almost always – there’s a much simpler solution waiting to be
discovered, one that doesn’t involve all the architectural noise, convolutions
of the straightforward, and misguided emphasis on hooks and options for all
kinds of tangents which might be useful someday. Discovering that solution may
not be easy, but it is time well spent.

The Boy Scout Rule

My friend Dom Finn wrote a post To Fix Or Not To Fix. This is something I’ve thought about quite a lot as, lets be honest, everyone is maintaining a legacy codebase to some degree.

The Boy Scout (or Girl Guide) Rule is to leave the campsite tidier than you found it. This helps avoid making the problem any worse and the propagation of Broken Windows.

DDD 10 - 10 Practices

Among some of the more interesting feedback were some requests for some links and an overview of the things mentioned in my talk.

A few people mentioned they felt misled by the title and synopsis of the talk. I read it back and I think it’s accurate, but then I would. If you are one of those people can you email me at with suggestions on how I should change it. It would be greatly appreciated.


I’ve put them up on Slideshare though without the context of the talk they might not be much use. I used the Bangers font from Google web fonts and the tech light palette from colourlovers.

Versioning APIs Sucks

Also, versioning APIs sucks. It’s not that it’s hard, it’s that once you publish
an API, you pretty much have to support it forever. This is especially true when
your API is being consumed through multiple layers. I can’t force game
developers to upgrade, because they can’t force their users to upgrade. The
lesson here is simple, if you don’t have to make something public, don’t!

Software and Schrödinger’s Cat

An interesting article exploring the relationship between quantum physics and continuous delivery.

None of it is real until it is in the hands of actual users. I don’t mean
someone who will poke at it a bit or evaluate it. And I don’t mean a proxy who
will tell you if the users might like it. I mean someone who will use it for its
intended purpose as part of their normal routine. The experience those users
report is reality. Everything else is speculation.