Continuous Integration – The State of CI & CD in 2015

On December 14, 2015, Posted by , In Continuous Integration,DevOps Practices,Engineering, With 1 Comment

The state of Continuous Integration & adoption within the industry.

Implementing Continuous Integration [CI] at an enterprise software organization can be just as controversial as it is valuable. The CI engineering paradigm colloquially aspires to increase software release quality through frequent code integrations (merges), smaller batch sizes, and immediate feedback surrounding changes introduced to a software system. Arguably the most controversial segment of CI surrounds development personnel and their religious views of branching patterns. Developers can be fairly cranky when it comes to fundamental alterations in the way they work. Motivating strong-minded individuals who are introverted and shy to communicate more and expose risk earlier is no easy feat.

The concept of CI received numerous accolades and realized significant traction soon after it was introduced as a development best practice via extreme programming [XP]. Since its inception CI has been spearheaded and evangelized by well-known thought-leaders within the technology industry [Martin Fowler, Jez Humble and James Shore]. With as much press as CI has received, software engineering organizations and technology providers have to this day seen mixed results in encouraging engineering teams to adopt continuous integration best practices. The main area of contention surrounds avoiding the urge to create isolated feature branches, or implement promiscuous integration [PI] practices. Since the invention of GIT, GITHUB and similar Distributed Version Control Systems [DVCS], CI branching strategies and best practices are now becoming an afterthought at enterprise software engineering organizations. CI has always aspired to become a foundational engineering best practice regardless of technology and platform constraints. Teams that practice CI properly have been shown to catch defects earlier in the software development lifecycle [SDLC], and release software with higher quality. So why then is there such contention within the industry?

While CI has increased in notoriety, SaaS providers and open-source software developers have introduced countless DevOps related automation solutions. So much so that there is even a periodic table, which details the numerous solutions [shown below, provided by Xebia labs] on the market today.

While there exists a multitude of CI software solutions, very few of them document CI best practices (branching, merging, code conflicts etc.) and show the value that these practices provide to engineering and businesses. Software providers are also advertising the various branching methodologies that their CI solutions support. Unfortunately this creates a rift between what was a once well-defined practice and what is now implemented at various software shops around the world. Some engineering professionals will even question your sanity when a proposal around mainline development surfaces. It is important to note that the tools these organizations have created do add ancillary technology support for Continuous Integration (CI), Continuous Delivery (CD), and Continuous Deployment practices; however they do not represent the fundamental CI engineering practice.

Web accessible documentation for CI software solutions includes a bare minimum amount of industry research on CI. Their advertising of bullet point features continues to drive their wares to the top of any web based search results. This is summarily due to a large marketing budget, which can almost always generate more backlinks and SEO gander than a blogger or book can. This problem seems to be compounded by the fact that less and less people read books and more and more people rely on quick Google results. The end result of this misappropriated research is confusion in the definitions surrounding CI, CD best practices.

The originating founder of Continuous Delivery aptly attempted to address some of this confusion in his 2007 Spark conference speech on Continuous Delivery. Jez stated:

“Continuous Integration is hard. It’s not running your CI tool on your feature branches. It’s not running your CI tool and having the build be red all the time; but no one cares. ” – Jez Humble

While his efforts and those of other CI ‘purists’ have not gone completely unnoticed, it’s now becoming increasingly difficult to differentiate tooling documentation for CI and documented research surrounding continuous practices as a engineering discipline. Let’s look at Google for an example. Via Google (aside from the Wikipedia article and a few blog posts) the first page of search results for “Continuous Integration” provides links to at least 5 CI product offerings. Within these results, each company selling a CI product provides a slightly different definition and approach to ‘CI’, which is tailored to fit their specific product. None of them seem to address the ‘integration’ practice required as part of continuous integration.

The ongoing confusion and mixed documentation leads many technology professionals to erroneously believe that CI is defined as the adoption of a tool OR simply represents auto-builds on feature branches. Modern DVCS technologies such as GIT have seemingly compounded this issue by providing easy feature branching mechanisms and have integrated code review practices into feature branches. This mendacious concept continues to spread throughout the ranks of engineering and software technology professionals, and is a reflection of the improper research conducted by marketing professionals. In essence the practical implementation requirements of Continuous Integration or Continuous Delivery seem get lost amidst the ocean of commoditized CI tools.

To help clarify the ambiguity surrounding CI and CD let’s take a quick look at the definitions of each.

Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies to a shared mainline several times a day. – Source: Wikipedia + Martin Fowler

Continuous Delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software faster and more frequently. 

Now that we have CI/CD and their respective definitions, let’s clarify the proposition a bit further. At this point I am going to draw a line in the sand. An organization cannot purport to be practicing CI with feature branches in play. This is because the ‘integration’ practice is missing and barriers have been erected that prevent the continuous flow of changes. Integration in the CI realm is synonymous with merging code and integrating work with the larger engineering team. In addition an organization cannot leverage continuous delivery unless continuous integration has been achieved first. This is by definition. Continuous Delivery is an extension of Continuous Integration and represents the furthering of CI practices to expand beyond simply merging and building code frequently. Into the realm of proving that the software is releasable through automation. The idea here is that many micro sized changes to a software system have greater value and less risk than obtuse and large commits that were created in isolation and outside of the realm of normal visibility.

Many engineers and technology professionals will argue that using many short-lived feature branches managed by a well-disciplined team represents the practice of CI. While this practice is better than creating long-lived feature branches it is more commonly known as promiscuous integration [PI]. The reasoning is that the practice needs to take into account the following points.

  1. The amount of time a feature branch will live
  2. The frequency in which the feature branch is merged into the mainline
  3. Possible syntactic or functional code collisions that are lurking in other feature branches being developed in parallel
  4. The delay in merge due to a longer than expected code review process
  5. The high frequency of communication and collaboration, which is required by CI
  6. The amount of drift between the feature branch and other parallel feature branches being worked on in comparison to the mainline (releasable entity).
  7. The encouragement and discipline required by mainline development to branch in code (using feature toggles or branching by abstraction) instead of in VCS.


The encouragement and discipline required by mainline development is probably the hardest aspect of CI. To successfully adopt and implement continuous integration and continuous delivery in an enterprise organization will require trust, the removal of process barriers and the creation of a blameless culture. These concepts are becoming more and more widespread as software engineering matures, complexity is traded for simplicity and releases are decoupled from deployment.

[Total: 3    Average: 4/5]
Continuous Integration – The State of CI & CD in 2015 written by jmcallister average rating 4/5 - 3 user ratings

One Comment so far:

  1. “Many engineers and technology professionals will argue that using many short-lived feature branches managed by a well-disciplined team represents the practice of CI”

    There is a fundamental misunderstanding of so called “feature branches” that led to its misuse and misapplication in Martin Fowler’s bliki (even tho it is now the prevalent use of the term in the post-agile + devops world of today).

    In the 90’s it was common for large system’s development that spanned multiple teams and 100+ developers (using Waterfall or V-model) to split up development across teams in one of two ways: by subsystem/component (resulting in subsystem/component teams each with their own mutually exclusive set of repositories); or by feature (resulting in feature-teams, often using feature-branches).

    A single (feature) team might have its own (feature) branch, but there was never supposed to be multiple feature-branches for a single team. Each-time was supposed to treat its team-branch as a kind of trunk/mainline and integrate to it as frequently as possible (at least daily), and then the full system integration/build was to run nightly (because doing a full+clean build of the entire system and running ALL of the [automated] tests back then would still take hours, whereas the “team-level” build needed to build less of the codebase (and run fewer tests) in order to get more rapid feedback to the development team.

    The folks who did this successfully made sure each team was following trunk/mainline-based development for the scope of work for that team, and that each team’s integration branch was integrated as frequently as feasible to the system-wide mainline/trunk.

    Where this fell apart is that too many folks who tried to copy the practice either neglected the frequent (at least daily) integration at the system-wide level, and/or they applied this practice to a single team trying to use multiple integration branches, rather than at most one per team. And the thing being called a “feature” was a very large core-capability of a system that would typically take a small-to-medium-sized team anywhere from 3-6months to complete (using incremental development+integration within a Waterfall/V-model lifecycle)

    The rest (as they say) is history. It was a very successful large-scale systems development practice for a large multi-team effort in order to do build+integrate as frequently as possible “at scale” when it was necessary to do at least two levels/tiers of integration in order to get rapid-feedback at smaller scale (for a single team) on their subset of the system while still needing to do full+clean build+test to close-out the larger feedback loop.

    These day’s, branching models like “GitFlow” are intended for a single team (large or small) and the thing called a “feature” is either a “topic branch” (e/g/, for a single story that takes 1-2 developers at most 1-2 weeks) or else is a “feature branch” that takes a small team 1-2 weeks to implement (and if they are an “Agile” feature-team, the idea is supposed to be not that they need a branch-per-feature per team, but that the entire team should “swarm” upon only one feature at a time before moving on to the next feature).

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">