Continuous Automation: DevOps • Agile • Leadership • Business Innovation: What the hell happened?
Continuous Automation: DevOps • Agile • Leadership • Business Recovering from Failure
Continuous Automation: DevOps • Agile • Leadership • Business You’re living in a buzzword world

QA Automation Architecture

Automated Quality Initiatives

One integral part of a Continuous Integration, Continuous Delivery or Continuous Automation is the successful implementation of automated testing. For Continuous Integration or Continuous Delivery to function automated testing MUST be part of the quality feedback loop. To illustrate lets consider the following SDLC diagram:


Over the course of my career I have seen only a few testing solutions implemented in a way where the organization realized the full value of the automation provided. I have spent the better part of a long while thinking about automated testing in relation to product releases.  What I have discovered is that the key to ANY automated quality assurance initiative is architecture and scalability. Before going too deep, lets define Architecture and Scalability:

Architecture: To plan and prepare a system or structure in such a way as to endure reasonably conceivable stresses

Scalability: To implement a solution in such a way to allow for natural progressive growth and avoid causing catastrophic failures

An automated testing nirvana is where the system catches defects efficiently, the defect catching automation system has little defects of its own, AND the defect automation system is architected properly and thought out carefully [go figure].

To emphasize tests cannot be just thrown into SVN and  executed in one large conglomerate pile, or written and never maintained. Just as important is that the test results cannot be isolated from the rest of the organization. As a software project evolves, so does careless architecture…

Test Definitions:

I have seen a lot of confusion in this realm. What defines a Unit test, a BVT or a Functional test? How do these come into play as part of a continuous delivery pipeline? Since there are a lot of methodologies and schools of thought  I thought I would research this subject and develop some logical conclusions:

Unit Test: An automated test created by a product developer which executes on the build machine to validate the functionalities of classes and methods.

Black Box Test: An automated test which executes against a visible entity. The test may access an API or a WebSite or the like but cannot however see into an environment’s file system or internals.

White Box Test: An automated test which executes alongside a visible entity. This test type CAN see inside an environments file system, and possibly may see into the internals of an application.

Smoke Test / Basic Verification Test (BVT): A set of tests that executes against a completed build to validate sanity of the system before proceeding to more in depth testing. These tests should be kept to a minimum and should not have an excessive runtime.

Functional Tests: An in depth suite of tests to validate a system in its entirety, divided into modules based on components and system architectures.

Why Do Test Suites Decay?

Automated tests can decay over the lifecycle of a product. Test code decays the same way production code decays when ignored or forgotten about. To avoid test decay… Test suites should receive the same attention as production code. Test code should be logical and structured and written with an expressed intent.  The code should live in the same environment structures as production code and failures MUST be addressed quickly.

Continuous Integration Feedback Loop:

As part of a continuous integration feedback loop automated testing is critical. However to keep the CI loop from creating throughput traffic jams it is necessary to keep automated test execution time reasonable within each level of acceptance verification.

Example: If the organization has 100 commits by developers a day and the automated testing suite takes 2 hours to complete, each developer can expect feedback in about 5 days when using a single  delivery pipeline.

Note: The above can be made a bit more reasonable if the organization implements a scalable build farm with a multinode testing harness and provides a scalable model for increasing the throughput. However the point is still valid.

To solve the throughput traffic jam problem there are a few guidelines an organization can employ:

  • Create the automation infrastructure to allow for multiple test runs simultaneously (a build farm approach)
  • Create the test suites to execute and validate in an expansive manner (minimally verification, moderate validation, extensive verification)
  • Use a defined set of basic quality test sets to determine initial viability of the build (Smoke/BVT)

To illustrate one possible Continuous Integration cycle. One might consider the following workflow:

  1. Developer commits code
  2. CI System creates a build,
  3. The CI system executes Unit Tests +  Fails / Passes the build based on results
  4. The CI system packages the build and deploys it onto a clean test device
  5. An initial set of Smoke/BVT tests are executed and the results determine Pass/Fail of the build.
  6. A more in depth set of functional tests execute on a grid style system

Continuous Delivery Feedback Loop:

Continuous Delivery represents a complete end-to-end feedback loop. CD includes feedback from the end consumer of the software platform as well as business stakeholders. Continuous Delivery by nature includes stop gaps in the delivery train to allow for manual verification or sign-off. These stop gaps are to enforce higher quality automated  releases to manufacturing and/or production.

To better illustrate the difference between CI and CD consider the following model:


As you can see above Continuous Delivery not only incorporates Continuous Integration but it extends the concept to the include final delivery to and feedback from customers. When considering implementing a Continuous Delivery pipeline consider the size of the organization, any silos that may exist and ways to bring everyone onto the same page.

Some guidelines and best practices to follow when implementing an automated Continuous Delivery solution:

  1. Audit the code base regularly to ensure your not shipping stale code or outdated / deprecated components
  2. Version stamp everything (including test suites) and use versioning to differentiate one build package from another
  3. Execute Unit tests as part of the build process and enforce the maintenance of them to ensure they are up-to-date
  4. Implement a binary management system to allow for archival of build artifacts and packages
  5. Implement stakeholder gated promotion based deployments to clean environments
  6. Use testing verifications as part of promotion which pass / fail the promotion based on quality metrics
  7. Continuously maintain the automated test suites to match them against the products functionality requirements
  8. Raise the quality bar as the software gets closer to production or manufacturing
  9. Execute automated test suites against production on a regular basis
  10. Standardize and automate everything (builds, deployments, test execution, notification of test results, etc.)


Continuous Integration, Continuous Delivery and Automated Testing are interdependent. Without automated testing Continuous Integration, and Continuous Delivery cannot be achieved. As such without a formalized plan for implementing automated testing in a scalable manner an organization cannot realize the benefits of such automation. A final narrative diagram shows this whole system in flow:



[Total: 0    Average: 0/5]

Category: Engineering

Share this Article

One comment

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

Article by: jmcallister