Throughout this series on How to Improve your DevOps Implementation, we have considered how organizations can left-shift security fully into the project design phase and how organizations can establish genuine cross-functional collaboration. In this 3rd of 7 parts, we can now consider how organizations can improve their DevOps quality assurance.
In a waterfall environment, quality assurance (QA) will typically be handled by a separate team responsible for ensuring that the application is right (satisfies the given business requirements) and is built right (is relatively bug-free). Thus, QA is often seen as a separate process scheduled after the developers finish their coding and build and before the project is handed off to operations for deployment. The process will include some automation. However, QA will often be dominated by manual effort.
Waterfall methodologies work well enough when project speed and the volume of throughput are not priorities. However, most organizations today acutely feel the market pressure to get functional products to market before their competitors. The fundamental concept embodied by the DevOps movement is enabling organizations to increase their throughput of functional products while decreasing the time of delivery.
Ideally, organizations mature from months to deliver a product to continuous delivery of every product. The challenge, though, which many organizations stumble upon as they work toward continuous delivery is that the traditional QA process cannot keep up with the increased volume, nor can it readily benefit from the same methodologies and tools that work so well for the integrated development and operations teams. In essence, DevOps quality assurance becomes a bottleneck in the pipeline[1] or, worse yet, is forced to take a back seat in favor of speedy and throughput. [2]
Increasing application throughput while deemphasizing the QA function is a short-sighted strategy. Equally true, it is unfeasible to allow a relatively slow DevOps quality assurance process to result in an ever-increasing backlog of issues in a CI / CD environment. Since there is neither time nor bandwidth to keep QA as a separate bailiwick in the pipeline, everyone needs to participate in QA.
Most organizations looking at or working with a DevOps production environment understand this. The issue is tactical. How can the organization transition to an up-front QA in a scalable, practical manner that does not slow down or hinder team production capacity?
Continuous Testing is not DevOps Quality Assurance
In a production environment dominated by functional silos, each team concentrates on and provides only its area of expertise to the final product. Developers provide the coding but leave QA and security to others. Security teams offer input on limiting attacker misuse-cases, but they do not worry about ensuring the product meets the specified business requirements. That is the task of QA teams, who also check that the product is bug-free.
If either security or QA throws up a red flag, the product goes back to development for rework. Only when everything is approved is the product “thrown over the wall” to operations for deployment – and the inevitable discovery of a slew of additional security and QA concerns.
No wonder it could take months to deploy a single application.
The DevOps movement seeks to sweep away the inefficiencies and bottlenecks of the functional silos. Since QA is an intermediary step between coding and deployment, integrating development and operations should, implicitly, integrate the intermediary steps as well. Unfortunately, realizing this integration is difficult. The tools and processes that work well for the developers and administrators do little to integrate and add efficiencies to QA and security.
It might, therefore, be tempting to sweep away traditional QA in favor of some form of fully automated process. However, it should be remembered that integrating development and operations required a culture change involving more than making developers and administrators share the same workspace. DevOps is a fundamental paradigm shift in the way the production pipeline works. Diverse functional priorities have to be aligned around a new, overarching goal. Within the DevOps movement, that goal is always to increase the throughput of functional projects – which is realized by moving each function as far left as possible in the production pipeline.
From a traditional perspective, up-front DevOps quality assurance is impossible since there’s nothing to test for quality until something has been built. Attempts to left-shift and automate at least some of the QA work in a CI/CD environment, though, naturally brings to mind continuous testing. Automation is a central theme in continuous testing. However, that is not the whole of it. Continuous testing also seeks to move to test earlier, make it faster, and test more frequently.[3]
Faster and more frequent quality testing in a CI / CD environment requires a certain level of automation, which has proven useful for detecting some types of issues and defects. Automated testings, though, have their shortcomings including an abundance of false positives, challenges to isolate and identify large system errors, and the persistent slowness to get bugs fixed compared to the speed at which they can be discovered. [4]
Furthermore, despite the speed and consistency afforded by automated testing, there are concerns which they cannot address. For example, automated tests cannot detect new bugs and vulnerabilities; they cannot keep themselves current as the project changes; nor can they eliminate all defects. Although some QA and security tasks can be automated, much about these functions requires involvement by subject matter experts.
Quality Assistance BEFORE Coding
The overarching goal of DevOps is to increase the throughput of functional products. Therefore, a DevOps implementation cannot be improved by attempting to automate QA fully or by dispensing with it. Some automation is possible and useful, but the input of experts is indispensable. The answer, preferably, is to transform the traditional quality assurance function into a left-shifted quality assistance function. [5]
From this perspective, it is entirely feasible to efficiently and effectively utilize up-front QA expertise in a way that results in increased DevOps speed and product throughput. The result will be quality “baked in” during the initial coding iteration. Projects will be built to the acceptance criteria specified, and the QA engineers will be able to ensure that any new tests required will be appropriately set up.
Up-front QA will allow developers to utilize their favorite issue tracking tool to simply check-off that the necessary criteria for acceptance are added to each unit before submission. This will enable the up-front QA team to monitor the implementation of acceptance criteria as the project progresses through the pipeline from the first unit tests all the way through staging environment dynamic testing.
Enable Up-Front Security
The same paradigm shift that yields up-front QA, enabling acceptance criteria to be “baked in” during the initial coding also enables “baking in” the necessary mitigating security controls. Fortunately, ThreatModeler is a single solution that provides up-front QA and security.
Architects automatically create detailed threat models – which can be customized to include QA considerations – when they whiteboard their projects as a process flow diagram in ThreatModeler. The threat models provide specific and immediately actionable outputs for fast-paced DevOps teams before they begin coding.
Having a discussion on specific DevOps quality assurance criteria and mitigating security controls with DevOps teams before coding helps the team circumvent bugs, defects, and vulnerabilities without the traditional rework or stockpiling issues for the next release. The result is functional products produced at the full capacity of which the team is capable.
Furthermore, DevOps teams can work with the actionable threat model outputs from within their favorite issue tracking toolset. ThreatModeler seamlessly integrates with JIRA and other CI / CD toolsets.
ThreatModeler can improve your organization’s implementation with up-front DevOps quality assurance. Ready to learn how? Click here to schedule a live demo today.
[1] Parizo, Christine. “Why Automation Doesn’t Necessarily Remove the Need for QA.” TechRepublic. CBS Interactive: San Francisco. February 23, 2017.
[2] Van Reijmersdal, Niels. “Unsolved Problems or Challenges in Software Testing.” StackExchange. Stack Exchange, Inc: New York. March 17, 2017.
[3] Lonn, Ragnar. “Continuous Testing: What exactly is it?” DevOps.com. Mediaops, LLC: Boca Raton. May 27, 2015.
[4] Reichert, Christopher. “4 Techniques to Reduce API Testing Errors and Improve your Automation.” Assertible. Assertible, Inc. April 5, 2017.
[5] Hunsberger, Ashlet. “Moving QA Upstream.” Sauce Labs. Sauce Labs, Inc: San Francisco. June 17, 2015.