Implement your contingency plan early

What I learned today — 13 March 2018

Today my team found ourselves in a difficult situation with a challenging release. We had a fix for a production issue waiting to be released, but releasing that without impacting the rest of the system was tricky. We decided that moving our scheduled release one day forward was the best course of action and we had a contingency plan to release the critical fix on its own if necessary.

As you probably guess, dear reader, it wasn’t that simple. An issue was discovered in the scheduled release and we raced to fix it, but that’s where we were tripped up. Not by completing that fix, which was simple enough, but by waiting too long before implementing the contingency plan. Failing to do so early enough created pressure and risk that we were eventually unwilling to accept and with everything ready, but with nothing released, we said that tomorrow will be another day. Had we implemented the contingency plan earlier we would have more choices and no stress.

Dear reader, don’t roll your eyes at me. We know that this kind of manual work to release software is no longer in vogue. We are still on a journey of automation and have already achieved much, but there are a few crucial pieces that’s still necessary, like building a dependency matrix into our deployment pipeline to ensure that deployments don’t break dependencies.