Lean principles are as relevant to software delivery today as they were to auto manufacturing in the 20th century. And when you apply lean practices to software delivery, you get the same great results — lower costs, faster cycle times and higher quality.
See the below infographic from Puppet to explore what software can learn from lean auto manufacturing, including the role DevOps practices and continuous delivery play in improving organizational performance, productivity and profitability.
There are two fundamental approaches of how to design automation processes, declarative (also often known as model) and imperative (also often known as workflow or procedural).
The purpose of this article is to explain the principles of each, explore a little of how they work, try to highlight some of the strengths and weaknesses and explore how they can be used together. I don’t think one size fits all—based on your requirements and some of what is explored below you may use a combination of the two, even if some of the tooling you use leans toward one way or the other.
The fundamental of the declarative model is that of desired state. The principle is that we declare, using a model, what a system should look like. The model is “data driven,” which allows data, in the form of attributes or variables, to be injected into the model at runtime to bring it to a desired state. A declarative process should not require the user to be aware of the current state; it will usually bring it to a required state using a concept known as idempotent.
What this basically means is that if you deploy version 10 of a component to a development environment and it is currently at version 8, changes 9 and 10 will be applied. If you deploy the same release to a test environment where version 5 is installed, changes 6 through 10 will be applied, and if you deploy it to a production system where it has never been deployed, it will apply changes 1 through 10. Each of the deployment processes brings it to the same state regardless of where they were initially. The user, therefore, does not need to be aware of the current state.
Once we have a model for our infrastructure or applications, any future changes required are made to the model. This means that as a model changes over time—and may have a current, past or future state. This raises the question of how can a model be applied to a target system, how can it know what changes to make? This is not always easy to answer and is not the same for every platform and runtime you are managing.
It’s hard to cover a broad subject like this without making some generalizations—and there are some in this section. There are broadly three methods used to keep an environment or system and its desired state models in line:
Maintain an inventory of what has been deployed. This where we maintain the state of what has been deployed, and what is in the desired state of a release, and only apply the difference.
For example:
Use Case: This type of state management is particularly useful for things such as databases or systems that requires updates from a propriety API where the state of the target cannot always be easily determined.
Validate/compare the desired state with what has been been deployed. As a simple illustration, we may want to determine if important_file.xml from my desired state deployment exists on the target system and whether it looks the same as the one in my package.
Use Case: Probably the most widely used implementation is to simply manage the files that many cloud-native runtimes use to store configuration definitions, such as ssh keys for a server or xml files to manage a java runtime such as tomcat. This also can be used to maintain runtimes that are managed via an API; for instance, What is my heap or config item for my WebSphere Server? Is this what it should be? If it is doing nothing, set it to correct desired state value.
Just make it so. There are occasions when we don’t really care about what exists. The most obvious example here is anything that is stateless, such as a container. There are probably very few example of why you would want to run any automation within a container to change its state; you would just want to instantiate a new one that has the new configuration you require.
The fundamental of what often is referred to as workflow, procedural or imperative approach is that a series of actions are executed in a specific order to achieve an outcome. For an application deployment, this is where the process of “how the application needs to be deployed” is defined, and a series of steps in the workflow are executed to deploy the entire application.
A standard example might include:
The often-cited criticism of this approach is that we end up with lots of separate workflows that layer changes onto our infrastructure and applications—and the relationship between these procedures are not maintained. The user, therefore, needs to be aware of the current state of a target before they know which workflow to execute. That means the principle of desired state/idempotent is hard to maintain—and each of the workflows are tightly coupled to the applications.
The reality here is that the situation is not black and white. Puppet is an example of what is seen as a Declarative automation tool, while Chef is said to be Imperative. Do they both support concepts of desired state and are idempotent? The answer is, of course, Yes. Is it possible to use a workflow tool to design a tightly coupled release that is not idempotent? The answer is, again, yes.
The benefit of using a workflow is that we are able to define relationship and dependencies for our units of automation—and orchestrate them together. In the example above, we can create a workflow to perform pre-install, installation and post-install steps, perhaps adding additional conditional steps if it’s in a particular environment such as production (disable a monitoring system, additional security configurations, etc.).
Procedural workflows also allow us to, for example, deploy component A & B on Server1, then deploy component C or Server2 and then continue to deploy D on Server1. This give us much greater control in orchestrating multi-component releases.
It is often suggested that the choice is between Imperative- and Declarative-Based Deployment. However, these two approaches are not mutually exclusive.
One of the issues in the world of DevOps is the lack of a consistent lexicon or terminology. Just ask someone who works in the industry what their definition of an environment is or what the term application means to them. My guess is you will get differing answers.
To me, an application is made up of components and a release is generally in the form of an application that is created from multiple versioned components. Now, depending on the organization you work in, you may or may not recognize the definition I have given—and if everything in your world is a stateless loosely coupled microservice on a cloud native platform, then this may be the case. If you work in a more traditional enterprise organization, you will more likely recognize the requirement for multi-component release orchestration, requiring procedural or workflow processes to handle the coordination and dependencies between the different components.
When it comes to deploying a specific component, I think it’s hard to argue against the benefits of using a Declarative or model-based approach. The concept of something being Idempotent—I can deploy any version to any target system if it has never been deployed to before, if we want to introduce a new change to an already existing environment or even if we want to roll a unit of automation back—is hard to argue against for its consistency and simplicity.
The future of IT Operations is doubtless based on things such as stateless microservices running in fungible containers, using Kubernetes watchers to trigger event-driven automation, much as the future of retail is for Amazon to deliver milk to my IoT-connected refrigerator using a drone.
That being said, most of us still live in the present and have the need, for some time to come, at least, to address some of the automation and orchestration challenges that exist in today’s IT landscape. With this in mind, a combination of Declarative (or model-driven) units of automation, coordinated by Imperative (or workflow-based) Orchestration can be used to achieve this. An imperative/workflow-based orchestrator also will allow you to executive not only declarative automation, but also autonomous imperative units of automation should you need to.
My recommendation is that an application workflow defines the order and dependencies of components being deployed. The Declarative Model for a component determines what action needs to be taken to bring a target system into compliance with the model.
It is hard to underestimate the importance of desired state—it has changed the way we look at system and application updates and drastically improved the visibility, audit and control of IT systems. Any progressive organization looking to move to a continuous delivery/deployment model or implement things such as autoscaling or “deploy and destroy” for development or testing purposes will benefit from the use of declarative/model-based automation.
However, a model-based approach does not answer all the difficult questions around orchestration. This, in my opinion (for the moment, at least) is best achieved using Imperative, Procedural or Workflows processes.
Dave Sayers is a co-founder and technical services director of MidVision, a provider of Release Automation and Continuous Delivery Software. He works closely with customers and partners on DevOps and Automation initiaitves throughout Europe and North America and is responsible for devising and managing the partner and channel strategy. Connect with him on LinkedIn.
You already know DevOps can make software delivery faster and more efficient. But did you know it can help to keep software more secure, too? Here’s how.
DevOps refers to a set of practices and cultural priorities related to the way software is designed, produced and deployed. It emphasizes collaboration between developers (the people who write code), IT Ops (the folks who manage software in production) and everyone in between—such as software testers and designers.
When done properly, DevOps greatly reduces the time it takes to bring software from idea to implementation to end user delivery. It also adds efficiency to the software delivery process in many ways. It allows different team members to work in parallel, for example. It also ensures that coding problems are found early in the delivery pipeline, when fixing them requires much less time and effort than it does once a bug has been pushed into production.
But making software delivery faster and less costly isn’t the only thing DevOps does. DevOps also helps make software more secure.
It does this in several ways. Consider the following:
DevOps isn’t a silver bullet when it comes to cybersecurity, of course. Keeping your systems and data secure requires careful planning on a number of fronts and implementing a variety of tools.
But DevOps—especially a highly automated, DevOps-based release process—should be one important weapon in your cybersecurity arsenal.
My recent blog, “7 Pillars of DevOps,” identified seven essential best practices areas that make up successful DevOps and explained that the pillars are not silos, but rather foundations that need to be kept in balance. One of the pillars is continuous testing (CT).
For well-balanced DevOps, CT performs tests that are fast enough to just fit within the time budget, yet thorough enough to cover the quality needed for deployment to production. Despite the need for balance, CT can be considered the true backbone of DevOps because testing is the one pillar that proves the validity of changes needed to operate and validate the correct operation of the other pillars from end to end across the pipeline.
Compared to older testing strategies in which large software changes were tested as a complete product after “release to QA,” DevOps CT strategically tests changes incrementally over a series of DevOps stages. CT begins the moment a change is made in development and continues through all DevOps pipeline stages. Release readiness is judged on analysis of test results accumulated prior to deployment. With Blue-Green, A/B and Canary test strategies, some tests may continue into the live environment after deployment, and the results may influence future CT activities.
Like continuous integration (CI), frequent, small, incremental testing has many merits. Quick feedback on problems enables a release consisting of many individual changes to be tested incrementally with confidence. The root cause of problems can be isolated much faster when the changes are tested incrementally. The problem is determining which tests are most relevant to run at each stage of a DevOps pipeline with content, with the risk of changes varying widely from one change to the next.
Given that testing is a stochastic sampling process, there is no such thing as 100 percent testing in any absolute sense. Left unchecked, the content of tests will grow exponentially beyond the content of the product that is being tested. No number of test resources, even if elastic on-demand cloud infrastructures are used, will keep up with the growing demands to satisfy test requirements. The key is to create and choose tests that are most relevant to the changes that are being processed in the pipeline rather than attempting to run all tests with no regard to actual matter what the changes you wish to inspect. A strategy that chooses the most relevant tests at each stage in the pipeline is optimal.
Several strategies for creating and selecting relevant tests are described below. These relevant test strategies may be used standalone for specific DevOps stages or a plurality of these relevant test strategies may be used end to end across the pipeline.
The above CT practices indicate that relevant testing strategies reduces the time for CT execution and reduces the infrastructure cost for CT, without sacrificing quality delivered to production.
What do you think? Do you agree with the above relevant continuous testing practices? Are there others you recommend?
While DevOps implemented with all seven pillars provides a strong foundation for long-term enterprise business improvements, it is important to understand DevOps is not an island. Enterprises implementing DevOps should be aware that DevOps interoperates with other IT systems and practices. Enterprises are well-advised to put in place governance policies that encourage the selection of tool-agnostic IT partners with solutions that best suit the needs of each unique enterprise and can integrate and evolve DevOps together with all their IT systems.
A changing landscape requires new processes
At the recent Symposium/ITxpo, Gartner shared a list of technologies and trends that will be important for organizations to embrace and prepare for in 2017. These trends, which are listed in the report, “Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption,” reinforce the breadth of technology that IT must support, roll out and engage with. This includes everything from the 100 million consumers who will shop using augmented reality to dealing with the roughly 21 billion endpoints expected to be in use in 2020.
The report also demonstrates how fast technology changes directions. For example, Gartner predicts that in 2019, 20 percent of brands are expected to abandon their mobile apps. “Many brands are finding that the level of adoption, customer engagement and return on investment (ROI) delivered by their mobile applications are significantly less than the expectations that underpinned their app investment,” the report notes. However, this doesn’t mean less work. IT departments continuously have shift gears to support new processes, news apps and new technologies, such as the Internet of Things (IoT) and the massive amount of data that it produces.
The dynamic needs of today’s businesses are driving organizations to increasingly adopt new technologies and applications that oftentimes don’t play nice with the existing set of infrastructure and legacy systems in place. So, while it’s clear that the technology landscape is changing, what isn’t so obvious is how IT and business decision makers should be adapting to an industry that is in constant motion.
To keep up, IT must implement agile methodologies that can adapt quickly to market changes. The DevOps mentality is making its way into many organizations and involves the development team creating, building and evolving products to ensure the organization is up to date, while the operations team confirms everything is running smoothly prior to a launch going live. This level of checks and balances isn’t just a nice to have, it’s a necessity—one mistake and the entire system can be brought down. Fostering a greater level of collaboration between development and operations makes it possible to efficiently and effectively build, test and correct new features and new products.
While the shift to a more collaborative environment between the Dev and Ops teams can take time, using modern IT automation solutions can make this transition easier. Such solutions automate many of the routine—and even not so routine—elements of the application release process within DevOps. Many of the actions necessary to move from code to production—not to mention maintenance, testing and cleanup tasks—can be taken off the backs of developers and operations, saving them time so they can focus on higher-level projects and reduce manual errors.
In 2012, Facebook announced it was increasing the speed at which it released code to twice a day, every day. As Chuck Rossi, director of Release Engineering at Facebook said, his team follows the advice “ship early and ship often.” Facebook continues to follow this model, and while the company certainly didn’t invent it, the model has caught on and is spreading among companies developing virtually every type of technology. The catch? It’s an incredibly time-intensive process and not every company has the resources necessary for a dedicated release team. Plus, a true shift to DevOps doesn’t happen overnight, especially since there is a ramp-up time for folks to get on board. For example, Phil Dibowitz, systems engineer at Facebook, told TechTarget that it “led its Chef team through a three-year process to rethink configuration management to scale on par with the web-scale company’s data centers.” This took three years, he said, “not because of the technological change, but because of the structural change.”
A slow transition to a new technology or new way of operating is a common occurrence because historically the production systems are cordoned off from development groups. While the Ops team has to ensure the rollout will meet certain standards, the Dev team wants to speed the process to begin the final rollout process as soon as possible. A happy medium exists, but if it isn’t found, things can go very wrong. Unfortunately, 2015 gave us two major examples of what can go wrong if products can’t speak to each other and rollouts are completed too early: Remember the New York Stock Exchange and United debacles?
This is where modern workload automation comes into play. Automation and solid change management allows IT to flexibly add on new technologies with automated processes for building, testing, upgrading and configuring IT—all while reducing the slack time between various steps. For instance, a workload automation solution should be able to trigger a build, deploy the build to the test environment, run the test suite and, assuming testing meets acceptable criteria, promote the build to production. If something does go wrong, necessary remediation can occur, and through monitoring and alerting capabilities, the required parties can be notified proactively.
Additionally, automation solutions with pre-built integrations make it easier for disparate applications to communicate, creating an end-to-end approach to workflow creation and maintenance. As a result, IT organizations can minimize the risk of business disruptions and achieve a more fluid technology delivery to the business.
Although it’s not a silver bullet that will solve all DevOps transition issues, automation does reduce manual intervention, which leads to fewer delays and errors and improves speed at which tasks are completed. However, beyond the technological aspects, there must be a clear understanding by both sides (the Dev and Ops group) on what their counterparts need—and this requires a high level of trust and coordination.
While the specifics around the DevOps methodology certainly will change, automation will remain a critical piece of the process. With IT automaton, DevOps teams are able to refocus on critical IT business and operational processes, thereby reducing delays, decreasing errors and improving the speed at which tasks are completed. Ultimately, automation makes it possible to increase the speed of product releases and enable companies to stay ahead in the age of unpredictable customer preferences and ever-changing technological developments.
Mehul Amin, a 21-year veteran of Advanced Systems Concepts Inc. (ASCI), previously held roles as product engineer and technical support manager and played an integral part in building the company’s workload automation solution, ActiveBatch. In his current role as director of Engineering, Mehul drives high-profile initiatives and leads both the Product Engineering and Release Engineering teams. Mehul received a master’s in computer science from the Stevens Institute of Technology.
In 2016, DevOps grew in popularity and use, new tools were introduced and enterprises across industries realized the influence and benefits derived from agile development. But there are even more developments to look forward to as enterprises engage DevOps teams further.
Scaling-out solutions, providing a baseline for a new cultural approach to problems and continuous improvement will continue to be priorities. But there are a few items that we’ll likely see come to the forefront in the new year. In fact, 2017 should be an even more exciting year for DevOps.
Here are my top four predictions for what we can expect this year:
Security was a perennial news event throughout 2016, and we can expect the same in 2017. However, DevOps teams likely will be challenged to build good security practices into code as a part of application development, rather than refining security post-facto. This will lead to DevOps, QA and operations bringing security teams deeper into the DevOps environment, if not outright incorporating them as a part of the team.
The foundations for this development were put in place in early 2016 as the DevSecOps model gained traction in cybersecurity circles throughout the year. In 2017, we’ll see DevSecOps teams build on these foundations. After all, every company is at risk for a data breach.
The convenience of containerization makes it one of the most desirable technologies for DevOps teams, and next year we’ll see it gain even greater use. It’s easy to understand why: Containerization solutions, such as Docker, grant teams the ability to provide a consistent environment from development to production. We may even see containerization move to non-production environments over the course of 2017 due to its portability across multicloud platforms. And that portability is critical not just for production purposes but for security as well.
DevOps will, almost by necessity, develop an on-demand model in 2017. Companies are realizing the stark demand for DevOps skills and they will be meted out according to internal needs and external demands.
In particular, we’ll likely see a lot of opportunity for DevOps in cloud implementation, since enterprises leverage solutions across cloud providers. With cloud provider “lock-in” still a real concern for many enterprises, any provider capable of operating cross-cloud development—thereby allowing customers to switch to a low-cost cloud provider—will be in high demand. This is good news for customers, as it allows them to finally benefit from the elastic nature of cloud pricing models.
Finally, enterprises will use DevOps across the enterprise. Thus far, large enterprises have been experimenting with DevOps in discreet projects, but they have started to plan for employing DevOps principles at enterprise-level, and we will see this roll out next year.
Thus far, DevOps has been focused on adopting and orchestrating technology and tools to build automation pipelines. But moving ahead, we’ll witness DevOps change its focus from building to executing. With this industry development and the shift within DevOps to security and containerization at an enterprise level, we’ll begin the true potential of DevOps.
Tan Moorthy is Head of Global Services for Application Development and Management at Infosys, a next-generation IT services provider. Connect with him on LinkedIn.
Management guru Peter Drucker once said, “If you can’t measure it, you can’t improve it.” I would take a step back and say, If you can’t measure it, you can’t get it off the ground. Many companies that I work with are amid digital transformation initiatives, such as the adoption of DevOps and continuous delivery. For these initiatives to gain momentum and acceptance within their organizations, they must be able to demonstrate quick time to value or risk being sidelined. And then, these organizations must be able to actualize their transformations and improve their processes—both enabled by the right metrics.
Indeed, one of the core pillars of DevOps culture—going all the way back to the first DevOpsDays event in 2010—is measurement. It is the M in the CAMS (Culture, Automation, Measurement, Sharing) acronym coined by Damon Edwards and John Willis.
Despite the prominent placement of measurement in the DevOps world, there is not really a lot of guidance about metrics in the common literature. For example, the original “Continuous Delivery” book by Jez Humble and David Farley devotes maybe a dozen pages out of 500 to it. And the recent “DevOps Handbook” by Gene Kim, et al is thin on the topic as well.
The coverage gap around measurement and metrics is driven primarily by the fact that the DevOps space is new and there are few standards around which to build an easily documented metrics structure. That leaves practitioners with the challenge of creating something they can use pragmatically within their own organizations. Anyone building a system of measurement for their organization must have a reasonable understanding of metrics as a discipline and a good understanding of the processes they intend to measure.
There are two types of metrics that I discussed in a recent session at CA World ’16 that are important when measuring a complex enterprise process.
Performance and result metrics are very closely linked: Performance metrics measure the effectiveness of the activities that drive the result metrics. For example, measuring the average number of deals closed per salesperson per period would be one measure of the effectiveness of a sales organization (a performance metric) which, in turn, would be a factor in driving revenue (a result metric).
While DevOps literature generally does not provide a lot of detail about metrics, it does provide advice on key result metrics. The primary one is lead time. Lead time is a powerful measurement that originated in Lean manufacturing. It originally dealt with how long it took a company to deliver a product to a customer after receiving an order. In DevOps terms, lead time summarizes the amount of time it takes to deliver one single change or feature from business idea through to running software in Production.
Lead time is calculated by aggregating the work time (referred to as cycle time) on a change plus the time when the change is in progress, but not being worked on (referred to as wait time). The formula that LEAD TIME = CYCLE TIME + WAIT TIME is a powerful tool for understanding the result metric level; however, the challenge in most enterprises is figuring out a workable system of supporting performance metrics.
You’ll find that the result metrics part of the effort—calculating the lead time for an application change—is relatively straightforward even when corporate structures can obscure exactly when work (cycle time) is happening versus when things are waiting (wait time). The Lean manufacturing world provides techniques for analyzing a flow.
A common technique is called value stream mapping (VSM). VSM provides guidance for breaking down a complex process into identifiable “steps” to provide measurement points throughout the process for cycle time and wait time. The measurement points then provide milestones for tracking the progress of the single change or feature from business idea through to production use.
For example, in an enterprise application development situation, typical application delivery pipelines provide several logical points that can be used as steps for VSM. Using those points, it is simple to measure linear time through the process. The common logical measurement points are:
The performance metrics that determine how efficiently cycle time and wait time are utilized, however, are significantly less obvious. Software development is iterative by nature. That means that individual steps in the value stream will be repeated many times and the cycle and wait time metrics ultimately will obscure the quality of what is happening at the measurement points due to natural tendencies to aggregate and average the values.
Establishing the performance metrics for a continuous delivery pipeline requires a bit of analysis. The performance metrics must be geared to understanding how to reduce the cycle time per repetition and the overall number of repetitions at each measurement point in the delivery pipeline. Here is a pragmatic, two-part pattern you can follow:
The first part is to break down the iterative pieces of the pipeline into logical ‘mini’ value streams. For example, each QA environment must be provisioned, configured, have changes deployed, and then validation activities occur. Those steps can provide measurement points and, more interestingly, can be used symmetrically in all environments in the pipeline enabling questions such as, “Does one take a disproportionate amount of time relative to the others?”
The second part is to look at the qualitative items that drive either long repetitions or increase the number of repetitions. For example, how many environments does a configuration package go through before rejection? More can actually be more expensive, slower and likely to be indicative of requirements problems, while fewer may indicate code quality problems. Another example might be smoke test failure rate in each environment. Variance in one environment could indicate problems in either the configuration management or deployment automation, but would require repetition of the deployment or manual correction. Finding that problem and fixing it ensures better cycle time at that step and stage, and serves to keep lead time down.
It may be a very long time before the DevOps movement reaches consensus on a system of metrics. The variability in software environments and industry needs means that it is unlikely for one size to fit all. However, with a pragmatic approach and some lessons from the Lean manufacturing world, it is possible to produce a workable and effective system of metrics for continuous delivery pipelines in the enterprise.
Vendors, such as my employer, CA Technologies, are helping by building more integrated systems of pipeline management tools to help shift from merely reporting metrics to applying them with modern analytics-driven techniques. The dividend of these approaches and tool improvements is that it is getting easier to gain a clear understanding of your process and to demonstrate to the business that the DevOps initiative is, in fact, worth the effort and investment.
No single cloud vendor has the perfect solution for your unique business situation. The only way to maximize the benefits of cloud services to your business is by dealing with several cloud providers and monitoring all of them simultaneously. That’s the IT equivalent of singing an aria while standing on your head and scrambling eggs. (Not really, but it can seem that way.)
Here are 12 things to keep in mind when planning your company’s multi-cloud strategy:
The potential complexity of multi-cloud management is shown in this flowchart encompassing the many elements that need to be stitched together precisely. Many of these low-level operations must be automated to optimize your strategy. Source: Carl P. Lehrmann, 451 Research
Focus on the apps in your organization that will benefit most from a cloud-native approach, and wait on those that are better served by deployment on a traditional data-center platform. Source: TechTarget
“Continuous” is one of the most popular words in the DevOps lexicon. What does it really mean, and what is the difference between continuous delivery, continuous deployment and continuous integration? Keep reading for some perspective.
Like many technological concepts, continuous integration, continuous delivery and continuous deployment are terms that DevOps teams tend to use without defining them precisely. For that reason, it’s worth taking a closer look at what each of these terms actually means.
Continuous integration can really mean one of two things, depending on the context.
In the narrower sense, continuous integration refers to the work done by a continuous integration server, such as Jenkins or Bamboo. A continuous integration server automatically tests code written by individual developers to make sure it can be merged into the main code base. By automating this process, continuous integration servers help developers to write and test small chunks of code on an ongoing basis, which minimizes the risk that a code change could cause serious problems and force developers to revert their application to an earlier state.
In a broader sense, continuous integration means the constant integration of changes to an application at all stages of the delivery chain. This includes but is not limited to the automated integration testing and code merging performed by an integration server. In this broader definition, continuous integration can include the integration of new design concepts to the delivery pipeline, for example. It also can involve other types of software testing beyond just automated integration tests.
In most definitions, continuous delivery refers to the process of delivering software updates to users on a nearly constant basis. Continuous delivery is made possible by continuous integration and other optimizations at earlier stages of the development process. But the definition of continuous delivery gets a little cloudy when you start comparing it to continuous deployment.
What’s the difference between continuous delivery and continuous deployment? For some DevOps teams, nothing. People tend to use these terms interchangeably. In fact, they are usually simply abbreviated as “CD,” leaving it up to the listener to decide what the speaker is actually talking about when he hears the term during a conversation about DevOps.
Some developers draw a distinction between continuous delivery and continuous deployment, however. For example, Mirco Hering writes that continuous delivery requires that the DevOps team manually release updates to users. In contrast, the continuous deployment pipeline is fully automated; users get updates as soon as they are written and tested, with no manual intervention by developers.
This distinction is a good way to think about the difference between the closely related concepts of continuous delivery and continuous deployment. It’s a distinction that will become increasingly important as technologies such as Docker make it easier to automate application deployment by allowing DevOps teams to drop new container images into production.
It’s worth mentioning, too, that the word “continuous” is usually something of an exaggeration when used by DevOps folks. It implies that software changes roll down the pipeline in a totally constant, never-stopping fashion.
That is an exaggeration. Integration, delivery and deployment are almost never completely continuous. In practice, a “continuously” integrated application is likely to be rebuilt and delivered something like every 24 hours, not every single time a code change reaches the end of the pipe.
That rhythm is still a lot faster, and closer to continuous, than the “waterfall” delivery pipelines that were common before the introduction of DevOps. But it’s still important to understand that, even with all of the automation tools available today, software development is rarely completely continuous.
The term DevOps implies that software delivery is all about integrating the development and IT operations teams. But the software testing team is an important, if oft-overlooked, part of the picture. Here’s why.
Could you deliver software continuously without involving the QA team in the continuous delivery pipeline? You could try, but it would probably not go well. Software testers play a vital role in continuous delivery, for the following reasons:
They are the glue connecting developers to IT Ops. If there is no one to test the code that developers write, IT Ops cannot deliver a stable product.
They help find problems before they become costly. If you skimp on integration, unit and performance tests, you won’t find problems in your software until after it reaches users. By that point, fixing those problems will be much more time-intensive than it would if they had been caught early on. Plus, your organization’s reputation will have taken a hit, because problems that could have been nipped in the bud prior to delivery ended up affecting users.
They know things that developers and IT Ops don’t. If DevOps is all about continuous visibility across the delivery chain, then it only works well when the delivery team’s collective expertise covers all parts of the delivery process. Since developers and IT Ops people are not experts in quality assurance or testing, you need to involve testing professionals in the continuous delivery chain so that they can help everyone else understand the parts of the process that QA professionals know in detail.
All of the above is to say that if the QA team is not plugged into every stage of the delivery process, your software either won’t be tested efficiently, or it won’t be tested at all. Not testing efficiently means that the QA team is not in sync with the rest of the organization. In this case, the QA team would delay software delivery because it would have to wait on developers and admins to complete its work, and vice versa.
Not testing at all, of course, is even worse. No one should deliver unvetted software to users.
The takeaway for organizations seeking to adopt DevOps practices is this: Even though the DevOps conversation on the surface may seem to be about only developers and IT Ops professionals, QA teams play an equally crucial role in continuous delivery. They need to be as involved in the continuous delivery pipeline as everyone else if an organization wishes to reap the full benefits of DevOps.