Author: Mike Vizard

  • Report: Software Quality Issues Costing U.S. Companies $2.08 Trillion

    Report: Software Quality Issues Costing U.S. Companies $2.08 Trillion

    A report published today by the Consortium for Information & Software Quality (CISQ) estimates poor software quality collectively cost companies in the U.S. an estimated $2.08 trillion in 2020.

    The report, co-sponsored by software quality and security tools provider Synopsys, attempts to calculate, at least theoretically, the cost of known software failures. The report identifies operational software failures, estimated to be $1.56 trillion, as the single largest cost. The next-largest cost contributor is unsuccessful development projects, which totals $260 billion, an estimated 46% increase since 2018, according to the report. The project failure rate remained steady, at about 19%, for over a decade, according to the report.

    Finally, legacy systems issues contributed $520 billion to costs in 2020, a significant decrease from the $635 billion these issues cost in 2018.

    That estimate does not include technical debt, which CISQ estimates at another $1.13 trillion. Technical debt costs have been growing at a rate of 14% per year since 2018, the report noted.

    Joe Jarzombek, director for government and critical infrastructure programs for Synopsys, said it’s apparent there’s not enough focus on quality assurance within software supply chains. However, as organizations begin to realize how dependent their digital business transformation initiatives are on quality software, business executives will be forced to shift focus.

    Unfortunately, most business executives today still don’t understand or appreciate how software is constructed. Most organizations don’t even have a bill of materials describing the components that make up an application, noted Jarzombek.

    That issue becomes especially problematic in the event of a security breach, because too many organizations can’t draw a correlation between vulnerabilities and the impact on specific components of an application, added Jarzombek. At the same time, however, cybercriminals are becoming more efficient at scanning applications to discover and exploit those vulnerabilities, said Jarzombek.

    Many organizations also don’t appreciate how adept cybercriminals have become at exploiting what are perceived as low-level vulnerabilities to distribute malware laterally across an application, said Jarzombek. There is a tendency to patch only the most critical vulnerabilities within an application, while the rest are left to future updates. When fixing low-level vulnerabilities is not prioritized, these vulnerabilities may remain over multiple release cycles.

    Ultimately, Jarzombek said, rather than treating security separately from the application development process, it should be addressed as part of a comprehensive approach to software quality assurance. The immediate challenge, however, is that not enough developers have access to the tools they need to address these ongoing cybersecurity issues.

    The impact of poor software quality on individual companies will vary widely, but poor software quality is itself a universal issue. Most software development teams are not deliberately ignoring these issues; instead, the processes currently employed to build software are not especially robust, mature or, for that matter, automated, leaving organizations vulnerable.

  • Orchestration Emerges as Crucial DevSecOps Enabler in 2021

    Orchestration Emerges as Crucial DevSecOps Enabler in 2021

    As DevSecOps best practices continue to evolve and mature in 2021, security orchestration is emerging as a cornerstone of a solid defense-in-depth approach to application security.

    DevSecOps assumes organizations will shift responsibility for application security further left toward developers. If this assumption is correct, then developers clearly need access to a range of code analysis tools to achieve that goal. The challenge they face is incorporating tools from multiple vendors within a DevSecOps workflow. At different points in the development process, it may make sense to employ a static application security tool (SAST) versus a dynamic application security tool (DAST) that stress-tests the application just before it is deployed.

    Not surprisingly, providers of these tools are starting to align with security orchestration engine providers. ZeroNorth, for example, recently added ShiftLeft, a provider of a SAST tool, to the list of security tools that can be orchestrated via its software-as-a-service (SaaS) platform.

    Joanne Godfrey, director of product marketing for ZeroNorth, said organizations that have embraced DevSecOps are standardizing on two to three security tools. This mitigates the risk of becoming overly dependent on a single tool to discover vulnerabilities, Godfrey said. No single security tool can uncover every possible vulnerability in a timely manner, so DevOps teams must work to integrate multiple security tools into their workflows.

    Developers need tools capable of identifying vulnerabilities as they write code. They should also construct tests to proactively flag vulnerabilities before those are incorporated into a larger build. New vulnerabilities, however, are discovered all the time and it’s not uncommon for a container build deemed safe yesterday to suddenly need an update as quickly as possible. DevOps teams also need to secure the runtimes and hosts on which application builds are deployed, which typically requires more collaboration with the security teams usually responsible for managing platform security and DevSecOps.

    Despite this need to collaborate closely, the cultural divide between DevOps and cybersecurity teams is often wide. While there’s a lot of interest in DevSecOps best practices, developers don’t always prioritize application security when there are delivery deadlines to be met. This often results in security issues being addressed late in the application development process or, worse, addressed after the fact as part of an update delivered after an application has been deployed in a production environment.

    As application development accelerates, it’s imperative that the current divide between DevOps and security teams is bridged. Short-handed cybersecurity teams simply can’t keep pace with the rate at which applications are being built, deployed and updated. The need to shift more application security responsibility left toward developers increases daily.

    The challenge now is, rather than simply delivering an empty DevSecOps sermon, IT leaders need to practice what they preach and provide developers with the tools and processes required to secure their applications.

  • Reserved.ai Applies Machine Learning to Rein in Cloud Costs

    Reserved.ai Applies Machine Learning to Rein in Cloud Costs

    Reserved.ai has added a new artificial intelligence (AI) tool to more accurately predict cloud computing workloads and rein in cloud spending. The company says its Purchase Planner tool uses machine learning to optimize cloud computing workloads by automatically purchasing cloud reserve instances based on the forecasts it generates.

    Purchase Planner leverages the machine learning algorithms Reserved.ai created to monitor and forecast when organizations should optimally reserve instances on a public cloud. The forecasts are based on customers’ current application workload requirements, as well as what the AI platform estimates overall demand for reserved instances is likely to be in the months ahead.

    The Purchase Planner tool is currently available on Amazon Web Services (AWS), and there is also a public beta edition available for Microsoft Azure clouds with support for Google Cloud Platform (GCP) planned for this quarter.

    Reserved.ai CEO Aran Khanna said Purchase Planner enables Reserved.ai to automatically purchase reserved instances on the AWS cloud. If, for any reason, those reserved instances are not required, Khanna said Reserved.ai will buy them back from customers.

    Given thriving demand for reserved instances on public cloud, Khanna said reselling those instances minimizes any financial risk for the company.

    Historically, many IT organizations have over-provisioned their cloud resources to mitigate the cost of having to purchase cloud instances on demand. In the wake of the economic downturn brought on by the COVID-19 pandemic, Khanna said more IT teams are attempting to optimize their cloud costs by reducing the number of reserved cloud instances they anticipate they’ll need based on their current cloud infrastructure consumption. Achieving that goal requires a massive amount of guesswork that can be more accurately calculated by machine learning algorithms, Khanna said.

    Khanna said the Reserved.ai platform provides the added benefit of requiring only 15 minutes to install, with a minimal infrastructure footprint required.

    Capacity optimization as an IT discipline has been around for decades. In the age of the cloud, however, IT organizations lost visibility into capacity utilization and cloud computing costs as developers spun up virtual machines. Now, finance teams require IT teams to, at the very least, maximize the number of workloads run while more accurately predicting cloud costs. Any actual reduction in cloud spending is a secondary benefit. The challenge today is that most cloud optimization efforts rely on the manual efforts of IT teams, and often, they’re unable to keep pace with the rapid rate of change in the cloud.

    Understanding the true cost of cloud computing will be more critical in the months ahead, as more organizations look to compare the cost of running one type of workload on the cloud versus another. Gaining visibility into those costs today will only be feasible when IT teams can rely on AI platforms capable of analyzing those costs in near real time. Otherwise, by the time an IT team becomes aware of an opportunity to reduce costs, the opportunity has already been seized by another organization.

    Regardless of how IT teams rein in cloud costs in the months ahead, the goal should be to maximize the number of workloads that can run without increasing the budget. After all, most organizations today are focused on maximizing their existing cloud investments to the fullest extent possible.

  • AWS Looks to Meld MLOps and DevOps

    AWS Looks to Meld MLOps and DevOps

    The rate at which machine learning operations (MLOps) and DevOps best practices are converging is poised for rapid acceleration in 2021. At the recent online AWS re:Invent conference, Amazon Web Services (AWS) announced it is adding a bevy of capabilities to Amazon SageMaker, a managed MLOps service, that includes a continuous integration/continuous delivery (CI/CD) service for MLOps it is calling Amazon SageMaker Pipelines.

    Amazon SageMaker Pipelines enables developers to define a machine learning workflow that encompasses everything from loading data to training configuration; algorithm set up and debugging algorithms. Rather than having to acquire and deploy a standalone CI/CD platform for building artificial intelligence (AI) models, AWS is making the case for using a managed service that enables developers to manage workflows from within the Amazon SageMaker Studio tools it already provides for building AI models.

    Workflows can be shared and reused between teams to either re-create a model or can serve as a launch point from which to create another AI model.

    Amazon SageMaker Pipelines also logs each event in Amazon SageMaker Experiments, which enables IT teams to organize and track machine learning experiments and versions of AI models. In addition, a Deep Profiling for Amazon SageMaker Debugger now makes it possible for developers to more quickly train AI models by automatically monitoring system resource utilization and generating alerts whenever bottlenecks are detected.

    AWS has also added a Distributed Training capability to Amazon SageMaker that it claims makes it possible to train large, complex deep learning models up to two times faster than current AWS approaches by enabling data and AI parallelism across its cloud services.

    The cloud services provider has also added an Amazon SageMaker Edge Manager service that enables developers to optimize, secure, monitor and maintain machine learning models deployed on fleets of edge computing devices. The Amazon SageMaker JumpStart service enables developers to search for AI models, algorithms and sample notebooks.

    Finally, AWS has added additional data preparation tools that come pre-configured with support of AWS and third-party data sources, a repository for storing AI models, a preview of additional tools for automatically matching AI models to data sources such as the Amazon RedShift data lake, the ability to match an AI model to a graph database and a tool that detects bias within AI models.

    Bratin Saha, vice president and general manager for machine learning services at AWS, said as usage of AI models increases, the scope of these initiatives naturally lends itself more to cloud platforms. Many applications are starting to employ multiple classes of AI models in combination with each other; Saha noted this requires massive amounts of data and compute resources to train those models that are only readily available in the cloud.

    Overall, AWS claims it already has more than 100,000 customers building AI applications on its platform using various tools and processor classes. AWS said in 2021 it will add G4ad Graphics Processing Unit (GPU) instances based on processors from AMD and revealed its intention to build its own GPUs next year. The company also plans to add an Amazon EC2 instance based on up to eight of Intel’s Habana GAUDI accelerators to deliver up to 40% better price-performance than current GPU-based EC2 instances.

    At the same time, AWS has extended its alliance with NVIDIA to make software modules for building AI models available on the AWS Marketplace.

    In effect, Saha said AWS is inviting IT teams to either build AI models on AWS or bring their own AI models, encapsulated in containers, to its platform.

    AWS is clearly aiming to dominate AI, regardless of how such capabilities are infused into enterprise applications. In fact, in many cases, AWS will automatically construct the AI model for customers, or simply return the results generated by an AI model via an application programming interface (API). It will be up to each organization to determine when building their own AI model makes the most sense, and to what degree they want to rely on AWS to automate the deployment process.

    What’s certain is that many of the same DevOps principles—created long before AI models ever came along—will still be relevant as the applications that use those models are continuously updated for years to come.

  • CentOS Rebellion Against Red Hat Gains Momentum

    CentOS Rebellion Against Red Hat Gains Momentum

    Two insurgent efforts to make sure the CentOS distribution remains viable have been launched following a decision by Red Hat to no longer invest in that fork of the Red Hat Enterprise Linux (RHEL) distribution.

    Launched in 2006, CentOS is widely employed by IT organizations that need an open source distribution of Linux to, for example, deploy a web server. Red Hat gained control over the CentOS trademark in 2014 as the primary sponsor of the project. Starting next year, Red Hat will only support CentOS Stream, which provides IT teams with a “rolling preview” of capabilities the company plans to make available on RHEL. Both RHEL and CentOS Stream are available via a subscription from Red Hat.

    The backlash to that announcement has been swift. Gregory Kurtzer, one of the co-founders of CentOS, has launched Rocky Linux with the support of more than 650 contributors to make sure IT organizations have access to a distribution of a fork of RHEL that doesn’t require a subscription.

    Kurtzer said many of those contributors work at independent software vendors (ISVs) that would prefer to see forks of RHEL continue to be available to their customers. Ultimately, the goal will be to add new capabilities to a Linux distribution in a more agile manner than what has occurred under the stewardship of Red Hat, added Kurtzer.

    At the same time, CloudLinux is pledging to spend $1 million annually to create a distribution of Linux dubbed Project Lenix that is also based on a fork of RHEL. CloudLinux has previously provided a hardened edition of CentOS that has been installed more than 200,000 times by 4,000-plus customers.

    CloudLinux CEO Igor Seletskiy noted the only other existing RHEL-compatible distribution of Linux is Oracle Linux. However, given Oracle’s stewardship of the open source MySQL and database, and the way the transition to open source editions of Java technologies has progressed, Seletskiy said there are many in the open source community who would prefer an alternative.

    As for the Rocky Linux effort, Seletskiy said Project Lenix not only will appear sooner but also IT teams will be able to convert entire fleets of servers over to it via a single command with no reinstallation or reboot required.

    Red Hat can do little to quell these initiatives because RHEL itself is based on open source code that anyone can employ to create a separate distribution. It’s not clear what impact this nascent rebellion will have on the ability of Red Hat to further monetize RHEL; however, it does illustrate how difficult it really is for a provider of any open source code to maintain control over the community that evolves around a given project.

    Many contributors to these projects are also becoming resentful of vendors that productize their efforts by converting open source software into, for example, a cloud service or subscription service that goes on to be the dominant means through which open source software is consumed. As a result, there is increased pressure for vendors to find a way to better reward contributors to these projects. Initiatives such as Rocky Linux and Project Lenix are clearly tapping into an egalitarian spirit within the open source community that runs counter to the way some vendors monetize these initiatives.

    It’s not clear to what degree these two efforts might inspire other revolts based on forks of open source code. However, it’s clear that the stewards of the primary open source project may not be the complete masters of their domain regardless of who may own the trademark for a specific distribution.

  • Copado Adds Value Stream Management to Salesforce DevOps Platform

    Copado Adds Value Stream Management to Salesforce DevOps Platform

    Copado, a provider of a DevOps platform for building applications that extend software-as-a-service (SaaS) applications from Salesforce, this week announced it has added value stream management capabilities.

    Andrew Leigh, chief marketing officer for Copado, said providing a platform that accelerates the development and deployment of application code is no longer sufficient. The winter 2021 release of the Copado DevOps 360 platform now includes value stream management capabilities that enable organizations to better prioritize DevOps projects, he said.

    In response to the COVID-19 pandemic more organizations than ever are now relying more on SaaS applications that can be accessed easily by employees working from home. IT organizations are also now using platforms such as Salesforce to jumpstart digital business transformation initiatives. Rather than write develop entire applications from the ground up, many organizations are employing low-code tools from Salesforce and others to extend, for example, customer relationship management (CRM) platforms.

    Leigh said most of the activity revolves around professional developers who are increasingly applying best DevOps practices to build those applications. Copado has certified more than 25,000 developers on its platform, he said.

    Given the current state of the economy, Leigh said interest in DevOps platforms is rising among organizations building Salesforce applications because they want to ensure a return on investment (ROI). Organizations that are investing six figures to build these applications are especially interested in value stream management tools that help make sure these projects are delivered on time.

    The value stream management capabilities added by Copado include support for the Scaled Agile Framework (SAFe) for building complex applications more efficiently. It also provides the ability to synchronize stories from Jira and Agile Planning boards.

    The latest version of the Copado DevOps 360 platform also adds a suite of declarative pipeline builders that allow non-technical teams to define and manage value pipelines without a single line of code. That capability is intended to simplify version control and automation flows to enable developers and administrators who manage Salesforce instances to better align their efforts.

    Copado added deeper integrations to third-party testing tools that incorporate artificial intelligence (AI) and updated its static code analysis and compliance tools.

    Salesforce is clearly not the only SaaS application platform that organizations are looking to extend, but it is among the fastest-growing. For its most recent third quarter, the company posted revenues of $5.42 billion, an increase of 20% year over year. The company is now lifting its future guidance to slightly exceed pre-pandemic levels. Salesforce is projecting fiscal 2021 revenue at the end of its fourth quarter will total $21.11 billion. Revenue guidance for fiscal 2022 has been raised to $25.5 billion. Salesforce earlier this week also announced its intention to acquire Slack, a provider of a messaging platform that is used widely by DevOps teams.

    Leigh said a dedicated DevOps platform for Salesforce applications is necessary because Salesforce applications are constructed using a set of objects and tables unique to the platform versus traditional application code. Regardless of the reason for requiring a dedicated DevOps platform, interest in building and deploying custom applications on top of the Salesforce platform has increased sharply since the start of the pandemic.

  • OpsRamp Makes AIOps Platform Simpler to Configure

    OpsRamp Makes AIOps Platform Simpler to Configure

    OpsRamp today took the wraps off an update to its namesake artificial intelligence platform for IT operations (AIOps) that makes it easier for IT organizations to declaratively configure the software-as-a-service (SaaS) platform.

    Ciaran Byrne, vice president of product management for OpsRamp, said the goal is to enable IT teams to embrace AIOps without having to hire a small army of consultants.

    The OpsRamp Fall 2020 Release adds a revamped user interface that provides access to a set of wizards that discover the assets that make up a specific IT environment and subsequently automate the configuration process.

    OpsRamp has also added curated dashboards that enable IT teams to launch queries using the Prometheus Query Language (PromQL), developed for the open source monitoring tool being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

    The latest edition of the AIOps platform also adds monitoring capabilities for containerized workloads and Kubernetes clusters. In addition, IT teams can monitor other cloud services including AWS ECS, Azure Functions, Azure Hyperscale (PostgreSQL) and Azure SQL Managed Instance.

    OpsRamp has also added IT process automation capabilities that can be triggered by alerts, updates to resources or on a recurring schedule. An OpsQ bot now also recommends automation workflows. Alternatively, IT teams can require manager approval before a process is launched.

    In general, Byrne said the goal is to improve the mean time to value for an AIOps platform managed by a team of data scientists who have extensive experience applying machine learning algorithms to IT processes.

    Efforts to automate IT have increased sharply now that most IT staff are working from home to help combat the COVID-19 pandemic. However, few organizations are able to set up and configure an AIOps platform in an on-premises IT environment on their own, especially now that outside consultants are no longer being invited to spend weeks working on projects, Byrne said, noting a SaaS-based approach makes it possible for IT organizations to take advantage of an AIOps platform while continuing to work from home.

    In general, the downturn in the economy has increased interest in AIOps because IT staff ranks are either remaining the same or shrinking even as IT environments continue to become more complex. Longer-term, Byrne said it’s not clear that hiring Level 1 and 2 IT administrators will be sustainable as AIOps platforms continue to automate processes.

    DevOps teams, of course, are especially keen to automate as many IT processes as possible, which is why there is a high correlation between organizations that have embraced DevOps and early adopters of AIOps.

    It may still be a while before AIOps is widely employed, but it’s now only a matter of time before machine learning algorithms are pervasive across IT. The challenge now is figuring out where the handoff between man and machine for managing IT ultimately will lie.

  • strongDM Extends Access Management as Code Efforts

    strongDM Extends Access Management as Code Efforts

    strongDM today announced it has added an application programming interface (API) and software development kits for Go, Java, Python, Ruby and other programming languages for involving the single sign-on (SSO) capabilities of its infrastructure access management platform, in addition to adding support for command-line interfaces (CLIs) exposed by cloud service providers.

    At the same time, strongDM has integrated its namesake platform with third-party secrets management tools such as Vault from HashiCorp as well as platforms such as PagerDuty’s incident management platform and Slack messaging tools.

    The access management platform can now also support any internal web applications and database platforms including DB2i, CitusDB, Oracle 11g, 12c, and 19c, Sybase ASE and IQ.

    Finally, the company has made enhancements to existing protocols including Mongo Atlas, SSH certificate-based authentication and SSH port forwarding, and added support for Windows remote desktop protocol (RDP) network-level authentication (NLA) and integration with other SSO tools such as Auth0 and VMware Identity One.

    Company CEO Elizabeth Zalman said strongDM, unlike other SSOs, is designed from the ground up to be incorporated within a developer’s workflow. The goal is to enable developers to manage access as code in the same way they manage infrastructure as code, she said.

    In general, access management is usually the last stage of automation to be addressed by DevOps teams. However, with most DevOps teams now working from home to help combat the spread of the COVID-19 pandemic, there has been a sharp rise in the number of organizations seeking to automate access management as an alternative to managing virtual private networking (VPN) software, said Zalman.

    Historically, many IT organizations have relied on Microsoft Active Directory (AD) or the Lightweight Directory Access Protocol (LDAP) to manage access. As an alternative, strongDM has been making a case for its platform since 2015. The company recently raised an additional $17 million in funding and claims to have more than 100 customers using its platform in production environments, including Peloton, SoFi and Betterment.

    It’s unclear whether DevOps teams are extending their existing workflows to include access management. However, as DevOps teams grow weary of using custom scripts to extend those workflows, Zalman said eventually they’ll look for a more consistent approach to automating access management. A major factor in that decision is the fact that most employees won’t be coming back to the office in the same numbers they did prior to the pandemic, even after a vaccine is discovered and distributed.

    At the same time, more DevOps teams are moving to embrace best DevSecOps practices, which inevitably will encompass access management. In fact, as part of that transition, it’s probable most DevOps teams will conclude the way access is currently managed is simply too antiquated to continue.

  • Mattermost Offers DevOps Command Center in the Cloud

    Mattermost Offers DevOps Command Center in the Cloud

    Mattermost today launched Mattermost Cloud, a software-as-a-service (SaaS) platform for managing DevOps workflows using pre-built playbooks that comes integrated with a wide variety of third-party developer tools.

    Chandar Venkataraman, executive vice president and chief product officer for Mattermost, said Mattermost Cloud enables IT teams to take advantage of a single-tenant DevOps platform that they don’t have to maintain in a way that forces developers to give up their preferred tools.

    In contrast, other DevOps platforms that are managed as a service require developers to make use of a narrow range of tools provided by the platform vendor, he said.

    Built on top of an instance of Kubernetes, the single-tenant capability also ensures that any data or artifacts stored on the platform remain private. The Mattermost Cloud Professional edition, available today, provides a full set of collaboration and DevOps features, while the Mattermost Cloud Enterprise edition, currently in beta, adds dedicated single-tenant cloud infrastructure, enterprise-grade secure network topology and strict data residency options.

    By building the platform on Kubernetes, he noted, it becomes much easier to scale the Mattermost Cloud platform up and down as required.

    Mattermost Cloud is essentially a DevOps command center that provides single pane-of-glass pre-built workflows that address most common DevOps processes. IT teams can add their own custom workflows as their DevOps processes mature, Venkataraman said, noting plugins for Jira, Opsgenie, PagerDuty, GitLab, GitHub and Jenkins are already available.

    The cloud platform also includes a pre-built best-practices incident management playbook to enable DevOps teams to address issues as they arise rather than trying to resolve them using a legacy ticket-based approach to IT service management (ITSM), he added.

    Venkataraman noted Mattermost doesn’t expect DevOps processes to supplant existing ITSM platforms based on ITIL-frameworks. However, many DevOps teams are looking for ways to resolve incidents within their existing workflows using, for example, a messaging platform versus a workflow designed for IT support staff.

    In general, Venkataraman said, many organizations that are relatively new to DevOps don’t want to manage the underlying platform. At the same time, he noted there are also many IT organizations that want to allocate more resources to writing code versus managing a DevOps platform.

    Each IT organization will need to decide when it makes sense to stand up their own platform for managing DevOps workflows and playbooks instead of building and maintaining their own control plane. Many of those IT teams are already making use of artifact repositories and continuous integration/continuous delivery (CI/CD) platforms that are accessed via the cloud, so a DevOps command center is only one more logical extension.

    Just as importantly, isolating the DevOps command center from the rest of the underlying DevOps platforms ultimately makes it easier for IT teams to switch out tools and platforms over time as they best see fit. The challenge now may be in finding a way to decouple workflows that are deeply intertwined with a specific platform.

  • CircleCI Report Finds DevOps Teams Adjusting to New COVID-19 Normal

    CircleCI Report Finds DevOps Teams Adjusting to New COVID-19 Normal

    An annual State of Software Delivery report published today by CircleCI suggests that after a spike in developer activity last spring at the beginning of the COVID-19 pandemic, the pace at which code is moving through a continuous integration/continuous delivery (CI/CD) platform has returned to normal levels even as developers continue to work from home.

    CircleCI CEO Jim Rose said after any initial rush last spring, it appears maintaining that level of productivity is simply unsustainable for most organizations. If heroic measures are continuously required, Rose noted it’s only a matter of time before DevOps teams start to burn out.

    Based on an analysis of 55 million data points gathered from 160,000 projects spanning more than 44,000 organizations, the CircleCI report finds that despite the ability of a handful of organizations that can deploy code multiple times per day, most organizations on average run workflows less than once (0.7 times) per day.

    However, Rose noted, there is too much focus on the number of times per day code is being deployed. The more important metric is duration. Half of the 11 million workflows measured by CircleCI finished in less than four minutes. Three-quarters finished in less than 11 minutes, with 95% of all workflows measured finishing in less than 35 minutes.

    Rose said durations are longer than in previous years, which suggests more testing is being included in workflows. In an ideal world, Rose said workflow durations should be in the range of five to 10 minutes.

    The report also finds 50% of the workflows went from failing to being successfully deployed in 55 minutes on average, with the median success rate 61% of workflows. Splitting the workflows by the ones run on default branches versus the ones run on non-default provides more insight, and the median rates are 80% for the default branch and 58% for the non-default branch. Failure rates for the non-default branch are higher because that’s the branch most developers employ to experiment in a way that doesn’t impact the primary workflow.

    Rose said the report also suggests there is a correlation between larger team size and better overall engineering performance. In times of uncertainty, IT leaders should consider increasing the size of teams to reduce burnout and maximize flexibility. Depending on the scope of the project, Rose said the CircleCI data shows that somewhere between five and 20 code contributors is the optimal number of team members. IT organizations would also be well-advised to consider how those team members are geographically distributed to maximize the number of hours in a day any project is being worked on without having to overwhelm a small number of developers working in the same time zone.

    As has always been the case, DevOps success over the long term requires organizations to balance multiple factors and metrics. After all, the primary goal for most organizations is not so much to increase the amount of code being deployed as much as it is to increase the quality of the code that ultimately gets deployed in a production environment.