CI/CD Pipeline Sprawl
Teams automate everything in the CI/CD pipeline — linting, testing, security scanning, dependency checks, container builds, deployments, notifications. Each addition seems reasonable. But pipelines accumulate steps over years without pruning. A pipeline that took 5 minutes in year one takes 45 minutes in year three. Flaky tests that fail 2% of the time become blockers when you have 500 tests. Security scanners produce hundreds of findings that nobody triages. The pipeline becomes the bottleneck it was designed to eliminate. Developers start working around it — pushing directly to main, skipping checks, batching changes to avoid waiting. The automation that was supposed to increase confidence becomes a tax on velocity that teams actively circumvent.
What people believe
“Automating everything in the pipeline improves quality and speed.”
| Metric | Before | After | Delta |
|---|---|---|---|
| Pipeline execution time | 5 min | 45 min | +800% |
| Developer wait time per day | 10 min | 90 min | +800% |
| Pipeline bypass rate | 0% | 15-25% of changes | +20% |
| Bugs caught by pipeline | Increasing | Plateaus as trust erodes | Diminishing returns |
Don't If
- •Your pipeline takes more than 15 minutes for the common case
- •Your team regularly retries pipelines due to flaky tests
If You Must
- 1.Set a hard time budget for pipeline execution and enforce it
- 2.Quarantine flaky tests immediately — fix or delete within a week
- 3.Run expensive checks (security scans, E2E tests) on merge, not on every commit
- 4.Review and prune pipeline steps quarterly
Alternatives
- Tiered pipeline — Fast checks on commit, thorough checks on merge to main
- Affected-only testing — Run only tests affected by changed files
- Local pre-commit hooks — Catch common issues before they hit CI
This analysis is wrong if:
- Teams with comprehensive CI/CD pipelines maintain execution times under 15 minutes over 3+ years
- Adding more pipeline steps consistently improves bug detection rates without diminishing returns
- Developers never bypass CI/CD pipelines regardless of execution time
- 1.CircleCI State of Software Delivery Report
Data showing average CI pipeline times increased 3x over 3 years across their platform
- 2.Google Engineering: Flaky Tests at Scale
Google's internal data on flaky test impact — 16% of their tests exhibit flakiness
- 3.Thoughtworks Technology Radar: Pipeline as Code
Recommendations for managing pipeline complexity as automation grows
- 4.DORA State of DevOps Report
Research showing pipeline speed directly correlates with deployment frequency and team performance
This is a mirror — it shows what's already true.
Want to surface the hidden consequences of your engineering decisions?