My blog, “7 Pillars of DevOps,” identified seven essential best practices areas that make up successful DevOps. One of the seven pillars is continuous integration (CI). The blog explained that the pillars are not silos, but rather foundations that need to be kept in balance. A well-balanced CI implementation performs integrations that are fast enough to fit within the time budget for an end-to-end DevOps pipeline.

Compared to older waterfall processes in which large software changes were integrated occasionally, the DevOps CI concept of frequent, small, incremental integrations has many merits. Quick feedback on frequent integration problems enables a release consisting of many individual changes to be built incrementally with confidence. The root cause of integration problems can be isolated much faster when the changes are integrated incrementally. Since faster integrations are better, just how continuous can integration be?

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

Contrary to the name, integration is never truly continuous. Even if a continuous stream of changes to integrate existed, typical CI process steps (listed below) require a non-zero amount of time. These steps absolutely interrupt the continuous integration cycle to some extent.

• Merge changes in the integration repository
• Check integration repository entry criterion
• Run static analysis scans of the integrated source code
• Orchestrate the build resources
• Create the build
• Save build the artifacts to artifact repository
• Run unit tests
• Save unit test results to artifact repository
• Release the build resources back to the pool
• Orchestrate the functional build test resources
• Run functional tests
• Perform analysis and assessments
• Save functional test results to the artifact repository
• Remediate any problems discovered during the above steps
• Promote the build to the delivery stage
• Release functional build test resources back to the pool.

For CI to perform as continuously as possible, there are three categories of CI best practices:

1. Modularize the product design to support modular CI processing

• Modular code designs, using 12-factor-app principles, allow incremental changes to be integrated.
• Microservice architectures enable each portion of a product to be integrated as available rather than requiring the entire product to be integrated at once.
• Using containers can help to optimize infrastructure orchestration.
• Re-factoring build schedules to remove any unnecessary build dependencies between modules or microservices eliminates time delays.
• All changes are checked into a common trunk branch. This requires the ability to toggle feature changes on and off quickly and easily to enable a release whenever it is needed.

1. Optimize the CI Infrastructure

• Vertical and horizontal scaling of both the CI build resources and test resources creates an infrastructure optimized for each CI stage run.
• Predictive orchestration of CI infrastructures enables CI and test resources and processes to be set up in advance.

1. Accelerate CI processes

• Pretest and pre-check integration deliverables prior to integration to trunk, to minimize costly remediation delays.
• Pipeline CI build and test processes to eliminate setup delays between processes.
• Use Fail fast to prioritize the most important tests as early as possible.
• Use accelerated testing techniques such as risk-based specific test selection to speed up CI testing.
• Use in-process analytics that can detect threshold exceptions prior to the end of the CI build or test process.
• Remediate problems fast using automated rules for roll-backs or targeted problem reporting.
• Establish a policy for deciding when and how to revert changes that break CI processes so that the pipeline doesn’t get stuck when check-in breaks the trunk.

The above CI practices commonly reduce the time for CI from many hours to minutes per build. While minutes is still not truly continuous, for most practical purposes CI processing in minutes is fast enough to suit even the most aggressive end-to-end DevOps delivery pipeline goals.

Summary

What do you think? Do you agree with the above CI best practices? Are there others you recommend?

While DevOps implemented with all seven pillars provides a strong foundation for long-term enterprise business improvements, it is important to understand DevOps is not an island. Enterprises implementing DevOps should be aware that DevOps interoperates with other IT systems and practices. Enterprises are well-advised to put in place governance policies that encourage the selection of tool-agnostic IT partners with solutions that best suit the needs of each unique enterprise and can integrate and evolve DevOps together with all their IT systems.

— Marc Hornbeek