Product Releasing Pipeline in AWS

2019-02-21 by Tomas Tulka

Continuous delivery (CD) brings a lot of ideas essential for a modern software product deployment. In this article we discuss how to follow CD principles by building CD pipelines with an example in AWS.

Taking CD principles seriously we have to keep in mind following: CD pipeline is a stream of changes and stages (actions) act like a filter.

An example of a filter action is testing or a manual approval. When testing fails, the pipeline run is rejected and the delivery ends. Similarly, an approval is a barrier, allowing or permitting the pipeline to go on, rather than a trigger.

Consider a simple CD pipeline for a service delivery:

Service Pipeline

After a source change triggers a new pipeline instance, code is built and goes into a DEV stage. The stage consists of three actions: the stack is deployed, tested and approved eventually. The approval could be triggered automatically via an external process like some system testing or manually by a developer. No new changes go into the stage as long as the stage is not either approved or rejected. This prevents a newer commit from modifying the stage while testing it.

When a commit is approved in the DEV stage, it can go on into the QA stage. In this stage QA tests and validations are proceeded. The approval actions prevents again new changes to be deployed while still testing. If no bugs are found, the stage is approved and the commit goes into the PROD stage (production), where it is deployed.

Continuous Delivery Example Scenario

Consider the following scenario of a continuous delivery process:

  1. The first commit c1 was pushed. It triggers the pipeline and it's automatically built and deployed into the DEV stage.
  2. Integration tests run successfuly and the DEV stage was approved. The pipeline moves into another stage - QA - where the commit c1 is deployed.
  3. Another commit c2 was pushed. The pipeline is triggered, the commit is built and deployed in the DEV stage. The QA stage stays on the c1.
  4. The c2 was approved in the DEV stage. QA testers are still busy with the c1, status quo.
  5. A new commit c3 was pushed, built and deployed into the DEV stage. QA testers still have no results from the c1.
  6. The QA testers found a bug in the c1 and the commit was rejected in the QA stage. The pipeline goes on and the next commit c2 is deployed into the QA stage.
  7. A new commit c4 was pushed, built and deployed into the DEV stage. QA testers are busy with the c2 now.
  8. Meanwhile the c4 was approved in the DEV stage and the next commit was deployed.
  9. A bug was found in the c4 already in the DEV stage - propably via a system/end-to-end/GUI test - the commit was rejected.
  10. Hardworking QA testers found a bug in the c2 as well and the commit was rejected. The pipeline goes on and the next commit c3 is deployed into the QA stage.
  11. After some time of testing, QA testers validated and approved the c3 and the commit was deployed into the PROD stage.

The following table summarize the above scenario:

Action Source+Building StagingDEV StagingQA StagingPROD Change
commit c1 c1 ✓ c1 ◯ - - c1 in DEV deployed
approve c1 in DEV c1 ✓ c1 ✓ c1 ◯ - c1 in QA deployed
commit c2 c2 ✓ c2 ◯ c1 ◯ - c2 in DEV deployed
approve c2 in DEV c2 ✓ c2 ✓ c1 ◯ - -
commit c3 c3 ✓ c3 ◯ c1 ◯ - c3 in DEV deployed
reject c1 in QA c3 ✓ c3 ◯ c2 ◯ - c2 in QA deployed
commit c4 c4 ✓ c3 ◯ c2 ◯ - -
approve c3 in DEV c4 ✓ c4 ◯ c2 ◯ - c4 in DEV deployed
reject c4 in DEV c4 ✓ c4 ✗ c2 ◯ - -
reject c2 in QA c4 ✓ c4 ✗ c3 ◯ - c3 in QA deployed
approve c3 in QA c4 ✓ c4 ✗ c3 ✓ c3 ✓ c3 in PROD deployed

Product Releasing Pipeline

Continuous deployment is a practice of deploying changes into the production automatically when developed, tested and validated via a DevOps team. This is not always the best idea, because a production release is actually a management decision.

A product is a group of services forming an independent unit, typically delivered under a single name. The product services should be tested, validated and released at once.

For such a scenario we would like to have a mechanism for approving all the related services together - a product releasing pipeline.

A product releasing pipeline typically consists of system/end-to-end/GUI testing and several manual approval steps.

Product Releasing Pipeline

When triggered, tests are run for the whole product (sytem) in the DEV environment. If the tests succeed, the changes are ready to go to the next stage - QA. This command action "stage into QA" is implemented as an approval of the previous (DEV) stage. Similarly, when QA testing is successful, changes are ready to be deployed into the production - implemented as an approval of the QA stage in service pipelines.

Example in AWS

AWS provides a fully managed CD pipeline solition: AWS CodePipeline.

The releasing pipeline is triggered via a CloudWatch event (for example every nicht at 1:00), end-to-end tests are executed and when succeed a Lambda is invoked. The Lambda puts an approval result programmatically into all the service pipelines.

Service pipeline names are prefixed with a product name - a parameter in the releasing pipeline. Service pipelines must follow this convention. Service pipeline stages must follow naming conventions as well (approve actions in the StagingDEV and StagingQA stages).

Approving the QA stage and releasing into the production is implemented via a manual approval action. Then the Lambda is invoked as well as in the previous step.

When a bug is found, the problem must be worked out individually - a problematic commit should be rejected in a particular service pipeline (the releasing pipeline don't reject delivery globally for all the service pipelines).

You can find a sample implementation of a releasing pipeline and two service pipelines in my GitHub.

Happy releasing!