Where is the psychological safety? Oh, wait, this is software engineering ...
The value of a CI/CD Pipeline is inversely proportional to how long the pipeline takes to run.
— Matt Rickard (@mattrickard) August 28, 2021
By definition, CI/CD pipelines are focused on two things:
- continuous integration (CI) – integrating developer changes into a single revision,
- continuous deployment (CD) takes that single revision and deploys it to customers.
CI/CD value is inversely linked to time because it can quickly become a Limiting Step in a developer's job. DevOps engineers will roll their eyes at this statement because the CI/CD pipeline is often their job. Instead, the right question to ask is:
Does a longer pipeline ultimately benefit the customer?
Take two extremes: no testing, which makes development initially fast but ships significantly broken features. Full testing, which covers every imaginable use case, but slows feature development to a halt (and still lets some bugs through).
Like reproducibility, software correctness is usually on a spectrum. Few, if any, large projects attain 100% test coverage, and they shouldn't. Integration and end-to-end tests are often comprehensive but can't cover all possible scenarios.
Extensive and long-running integration and end-to-end tests are good in theory, but have significant effects on the engineering organization and culture in practice.
We want bugs to be identified and fixed. Long-running tests can identify those bugs, but a lengthened feedback cycle works against that goal.
Faster feedback loops light a fire at both ends of the candle: bugs can be identified quicker, and developers can fix bugs quickly.
Failures can be difficult to debug in a long end-to-end pipeline. In bad (and common) cases, they can cause transient failures due to the pipeline itself. Test flakes reduce the trust in the pipeline, and an untrustworthy pipeline is not worth its weight in code.
The lesson is that testing is about risk management. Unit tests provide quick feedback and are easily debuggable. As projects scale, unit tests can scale alongside without issue. On the other hand, integration and e2e tests exponentially increase complexity as a project grows, especially in microservice architectures.
The takeaways:
- End-to-end and integration tests don't always benefit the customer and can be detrimental to the bug identification and fix feedback cycle.
- Most tests should be unit tests, followed by a smaller set of functionality tested by integration tests. Finally, the most critical and smallest subset tested as end-to-end tests.