Continuous delivery (CD) brings a lot of ideas essential for a modern software product deployment. In this article we discuss how to follow CD principles by building CD pipelines with an example in AWS.
Taking CD principles seriously we have to keep in mind following: CD pipeline is a stream of changes and stages (actions) act like a filter.
An example of a filter action is testing or a manual approval. When testing fails, the pipeline run is rejected and the delivery ends. Similarly, an approval is a barrier, allowing or permitting the pipeline to go on, rather than a trigger.
Consider a simple CD pipeline for a service delivery:
After a source change triggers a new pipeline instance, code is built and goes into a DEV stage. The stage consists of three actions: the stack is deployed, tested and approved eventually. The approval could be triggered automatically via an external process like some system testing or manually by a developer. No new changes go into the stage as long as the stage is not either approved or rejected. This prevents a newer commit from modifying the stage while testing it.
When a commit is approved in the DEV stage, it can go on into the QA stage. In this stage QA tests and validations are proceeded. The approval actions prevents again new changes to be deployed while still testing. If no bugs are found, the stage is approved and the commit goes into the PROD stage (production), where it is deployed.
Continuous Delivery Example Scenario
Consider the following scenario of a continuous delivery process:
- The first commit
c1was pushed. It triggers the pipeline and it's automatically built and deployed into the DEV stage.
- Integration tests run successfuly and the DEV stage was approved. The pipeline moves into another stage - QA - where the commit
- Another commit
c2was pushed. The pipeline is triggered, the commit is built and deployed in the DEV stage. The QA stage stays on the
c2was approved in the DEV stage. QA testers are still busy with the
c1, status quo.
- A new commit
c3was pushed, built and deployed into the DEV stage. QA testers still have no results from the
- The QA testers found a bug in the
c1and the commit was rejected in the QA stage. The pipeline goes on and the next commit
c2is deployed into the QA stage.
- A new commit
c4was pushed, built and deployed into the DEV stage. QA testers are busy with the
- Meanwhile the
c4was approved in the DEV stage and the next commit was deployed.
- A bug was found in the
c4already in the DEV stage - propably via a system/end-to-end/GUI test - the commit was rejected.
- Hardworking QA testers found a bug in the
c2as well and the commit was rejected. The pipeline goes on and the next commit
c3is deployed into the QA stage.
- After some time of testing, QA testers validated and approved the
c3and the commit was deployed into the PROD stage.
The following table summarize the above scenario:
|commit c1||c1 ✓||c1 ◯||-||-||c1 in DEV deployed|
|approve c1 in DEV||c1 ✓||c1 ✓||c1 ◯||-||c1 in QA deployed|
|commit c2||c2 ✓||c2 ◯||c1 ◯||-||c2 in DEV deployed|
|approve c2 in DEV||c2 ✓||c2 ✓||c1 ◯||-||-|
|commit c3||c3 ✓||c3 ◯||c1 ◯||-||c3 in DEV deployed|
|reject c1 in QA||c3 ✓||c3 ◯||c2 ◯||-||c2 in QA deployed|
|commit c4||c4 ✓||c3 ◯||c2 ◯||-||-|
|approve c3 in DEV||c4 ✓||c4 ◯||c2 ◯||-||c4 in DEV deployed|
|reject c4 in DEV||c4 ✓||c4 ✗||c2 ◯||-||-|
|reject c2 in QA||c4 ✓||c4 ✗||c3 ◯||-||c3 in QA deployed|
|approve c3 in QA||c4 ✓||c4 ✗||c3 ✓||c3 ✓||c3 in PROD deployed|
- ◯ - in progress
- ✓ - succeeded
- ✗ - rejected
Product Releasing Pipeline
Continuous deployment is a practice of deploying changes into the production automatically when developed, tested and validated via a DevOps team. This is not always the best idea, because a production release is actually a management decision.
A product is a group of services forming an independent unit, typically delivered under a single name. The product services should be tested, validated and released at once.
For such a scenario we would like to have a mechanism for approving all the related services together - a product releasing pipeline.
A product releasing pipeline typically consists of system/end-to-end/GUI testing and several manual approval steps.
When triggered, tests are run for the whole product (sytem) in the DEV environment. If the tests succeed, the changes are ready to go to the next stage - QA. This command action "stage into QA" is implemented as an approval of the previous (DEV) stage. Similarly, when QA testing is successful, changes are ready to be deployed into the production - implemented as an approval of the QA stage in service pipelines.
Example in AWS
AWS provides a fully managed CD pipeline solition: AWS CodePipeline.
The releasing pipeline is triggered via a CloudWatch event (for example every nicht at 1:00), end-to-end tests are executed and when succeed a Lambda is invoked. The Lambda puts an approval result programmatically into all the service pipelines.
Service pipeline names are prefixed with a product name - a parameter in the releasing pipeline. Service pipelines must follow this convention. Service pipeline stages must follow naming conventions as well (
approve actions in the
Approving the QA stage and releasing into the production is implemented via a manual approval action. Then the Lambda is invoked as well as in the previous step.
When a bug is found, the problem must be worked out individually - a problematic commit should be rejected in a particular service pipeline (the releasing pipeline don't reject delivery globally for all the service pipelines).
You can find a sample implementation of a releasing pipeline and two service pipelines in my GitHub.