Continuous software delivery in the digital age depends on CI/CD pipelines, which enable engineering teams to rapidly develop, test, and deploy code while maintaining high usability and consistency across environments. However, CI/CD pipelines — when systems start small and become complex — can themselves produce a source of friction. Pipelines set up to work for small projects have difficulty scaling with ever more repositories, larger test suites and larger development teams. Slow feedback cycles, increased infrastructure costs and decreased developer productivity are some by-products of bad pipeline design.
Here are 10 CI/CD pipeline mistakes that you’ll often come across as part of normal CI/CD pipeline activities in your own environment, based on common patterns found in a lot of engineering worlds.
Considering CI/CD Pipelines as ‘Set and Forget’
Some teams set up a CI/CD pipeline at the beginning of a project and then rarely touch it afterward. The pipeline of an application grows as it stacks up without a plan! Test suites get bigger, more build steps are introduced and repositories get bigger. However, without regular review, the pipelines slowly become slower and more complex.
Frequent monitoring to review pipeline performance should be a team responsibility. CI/CD pipelines should be reviewed and updated with the underlying software.
One Big Monolithic Pipeline
Another frequent mistake is to build a specific pipeline in charge of all the process of validation and deployment. Such pipelines frequently integrate: Linting, testing, packaging, security scanning and deployment. However, monolithic pipelines are slow and cannot be effectively maintained at scale.
Instead, the scalable route is to use distinct pipelines, designed for different purposes:
- Quick validation pipelines for pull requests
- Extended test pipelines for all merges
- Release deployment pipelines for releases
This structure allows developers to get prompt feedback and yet gain strong validation prior to production releases.
Inability to Monitor Performance of the Pipelines
Engineers put a lot of work into observing production systems; however, CI/CD pipelines can struggle to provide as much visibility. When pipelines slow down because they’re being taken down quickly or because they aren’t reliable, it can be difficult if not impossible for a team to answer questions such as:
- Which stage of the pipeline uses the most time?
- Which repositories trigger the most builds?
- Where do failures happen more often?
The addition of metrics, dashboards and logs to enable pipeline observability allows teams to identify performance bottlenecks and enhance pipeline stability.
Heavy Dependency Management is Responsible for Slow Build Times
Large projects can experience huge delays if they repeatedly download dependencies from external registries on pipelines.
Usage methods that improve dependency performance include: Dependency caching, Docker layer caching, artifact reuse between pipeline stages. Even a small step in caching can drastically reduce build time.
Balanced Testing is Often the Best
Too much testing or too few tests? The testing strategy has a direct effect on CI/CD pipeline performance. Some teams run the entire test suite on each commit, leading to lengthy feedback cycles. Others are not as good at CI testing, still need to do as much manual verification once they have CI.
It is recommended to use the following approach:
- Unit tests for every commit
- Integration tests performed on merges
- Full regression tests before release deployments
By this approach, developers can get quick feedback while maintaining software quality.
Ignoring Parallel Execution
Many pipelines will run tasks in this order because that was the default setup of the pipeline when it was created.
Modern CI/CD platforms support parallel execution of tasks such as running test suites, building microservices, performing validation checks. Teams can drastically reduce overall pipeline execution time and speed up development workflows through parallelism.
Creating Inconsistent Build Environments
Environment inconsistencies cause CI/CD pipeline failures most of the time. Build locally to one environment and CI on the other leads to avoidable unexpected breakdowns. This is when developers run builds locally in one environment and CI systems use another, and they find unexpected failures. Typically, that results in the common situation of code working in your local environment but failing on your pipeline.
A containerized environment or reproducible build environment will help keep the developer machine and CI systems consistent.
Resilient Pipelines Need to Improve
Builds can occasionally be interrupted by network instability, flaky tests or temporary dependency issues. Without resilient pipelines, there is a risk of failures being too frequent to prevent; instead, developers will only be forced to restart builds.
Retry logic for temporary failures, isolation of flaky tests and clear error reporting mechanisms can help in such situations. Enhanced pipeline resilience reduces unnecessary interruptions in an existing development workflow.
Poor Secrets Management Practices
Security is usually ignored early on in CI/CD build. Credentials might be stored directly in configuration files or environment variables as credentials (e.g., API tokens, service keys or database passwords) but without careful safeguards. When CI/CD pipelines share secrets with many services, including the infrastructure pieces, secure secret management may be quite critical.
It is possible to mitigate the security risk dramatically by implementing secret management tools and rotating passwords on-demand.
The Developer Experience is Neglected for Developers
CI/CD pipelines are developed to support developers and speed up the production of software to deliver it closer to market. When pipelines get slow, unpredictable or challenging to debug, they don’t just disrupt productivity, but they stall it.
Good pipelines do the following things: Fast feedback cycles, clear error messages, and predictable execution. When pipelines work well, developers rarely have to think about them.
Final Thoughts
CI/CD pipelines are often seen as support infrastructure, but they are essential to modern software delivery. Pipelines must evolve with the complexity of systems. Review of pipeline design, performance and reliability can lead to a dramatic increase in engineering productivity and system stability. Even the incremental benefits such as effective caching, better caching strategies, clearer pipeline stages and better observability, can create long-term benefits.
David Cramer is vice president of Product Management for the Cloud/DCA business unit at BMC. Prior to BMC, David was head of product management for CA Technologies. During his tenure at CA, David was responsible for application delivery, cloud management, virtualization and Infrastructure automation solutions. Before joining CA, David held executive positions at AlterPoint, Motive, NetSolve and Nortel Networks. David’s focus is on continuous delivery, cloud management and web-scale IT. David received his MBA from Southern Methodist University and a BS in Finance from Georgia State University. Connect with him on 
John Joseph, VP of marketing for 
