Big Bang Software Releases
REAL LIFE ENGINEERING
Your engineering group has been toiling away. You’ve made it from the last release to this release by adding features, implementing crafty bug fixes and churning the code base. Its release day, just shy of three weeks since your last release and the landscape of the software has changed. Its time to release and deploy it into production.
The OPS team fires up your software in production (may take them a few hours or more), and you head home for a long deserved weekend at the beach. Just as you sit to enjoy your well deserved time off your but the blasted phone starts going off. Clients have found catastrophic bugs in the system which have bubbled their way up to engineering for immediate hot-fix. Your boss is racing around trying to rally the troops and get them to fix these critical problems. The hotfix will almost certainly bypass a complete QA cycle, will often times be a hack to make it work, and will probably break every best practice in the book.. BUT management wants it NOW and something is wrong with the API, or the Core and your forced to pull an all nighter, which turns into an all weekender to put out the fires.
You burn your weekend toiling at the code, and cranking fixes in an attempt to salvage your Sunday evening. Finally you fix it. You come in Monday morning to the smoldering plethora of defects that still exist, and feel the grind sink back in.
ALLEVIATE RELEASE HELL
Releasing via a big bang events once every few weeks or worse every few months is a nightmare waiting to happen. For engineers this type of stress is unnecessary. How you say? Release your software more frequently! Make your software releases a non-event. Go a step further and design a system where EVERY build is a potential release candidate. If you release software changes frequently the changes to the code base are minor, and when something goes wrong (and it will), it will be obvious what was broken. The fix from here is easy.
As a supplement for more frequent releases its critical to automate as much of the release process as possible. Additionally, by adding automated testing to your QA cycles you can speed up the QA process even further.
The goal is to send releases out so fast that customers are almost always looking at the latest and greatest your team has to offer, and your release risks go way down. This provides additional business value of providing your company the ability to change things quickly with minimal impact based on customer feedback.
DELIVER VALUE INCREMENTALLY
When we focus on keeping the software releasable, usable, and release our changes frequently we understand the scope of the work we are doing and can make better decisions on what gets exposed to customers and when. So why is it we put ourselves through release hell? From my experience its partially culture based, partially feature driven, and partially embedded into the engineering mindset. Below are some key contributing factors:
- Programmers LOVE isolation (I think we would have gone into sales if we loved other people)
- Management loves to tell their boss that a feature is ‘complete’ and is usually measured on these ‘deliveries’
- Communication and coordination of work efforts is almost ALWAYS deficient when your tied to a computer all day
- Coordinating people to focus on a single mission without upsetting them is ALWAYS hard
- People are generally lazy and want the fastest path to completion
Please pick your favorite one above.
To address these issues we can look at what programming is at its core and build from there. Software development is NOT an art its a science. If we learn to use scientific method to develop, build, test and deliver we can define exactly what our end goal is. Ultimately reproducibility of our procedures is at the core of any science and we can leverage these practices to:
- Prove the software works by creating tests to validate the engineering hypothesis.
- Prove the software installs by writing automation to provision and deploy it.
- Prove that the customers like the feature or idea by collecting metrics to define business success
None of these ideas are new or even that magical. Its simply a matter of developing a level of self discipline to think it through, come to a common consensus within the organization to achieve a unified build, test, and delivery system.
To further this idea we should strive to when working on a new feature or major architectural change, keep the software releasable. How you say? I made major changes! Use a feature flip, or hide your code in the mainline until its ready. As software developers we utilize this technique more than any other pattern (maybe without even realizing it):
- IF THEN ELSE Conditionals
- BOOLEAN LOGIC
- OOP Instantiation
- Configuration files
All of the above development fundamentals are designed to fork the operation of the code in one direction or another. They expose or retract additional logic based on a criteria. We can most certainly apply this tactic to hiding an unfinished feature.
Software architectures are constantly being revised and improved. Sometimes the only option we have is to re-write whats already written. For long running features and architectural changes we need to manage the change effectively by employing branching by abstraction (see diagram), or feature flips.
We simply need to scope our work carefully and divide it into manageable chunks. If we can do this we can provide real value for the business and that provides us with our paychecks.