Testing Serverless Systems
Testing serverless systems is hard. In this article, we will discuss some practices which work well for my project.
Designing a system architecture is always about making tradeoffs. Microservices resp. serverless architecture has a lot of benefits, but some drawbacks as well. One of them is testing.
Serverless Systems
Because the view to serverless microservices can differ, let's start with a bit theory to define the terms we will use.
A serverless architecture is built upon managed elastic simple components. AWS offers for example components like Lambda functions, DynamoDB NoSQL database, SNS message service etc.
A microservice is my understanding a logical unit owning all the components to carry out its whole functionality. Microservices are built around a boundary context (domain-specific) and contains all the functions and resources to be able to run as an autonomous service. All the components are private for the microservice and the functionality is accessible only via an API. If you are adapting Infrastructure as code (and you should) you build a microservice as one stack (1:1). Typically you have an integration build pipeline per microservice. Microservices can communicate via domain events (preferred way) or API calls.
A fictitious on-line store can have for example these microservices:
Very important role in a serverless architecture have functions. Functions (like AWS Lambda or Google Cloud Functions) are the simplest deployable components, typically representing an action around an aggregate (what has to be processed within a transaction).
Applications are clients outside the system using the system functionality via API calls.
What and How to Test
The most problematic term in testing theories is a Unit Test, resp. the Unit. Some articles say it's the smallest functional block. Some add that a unit test must run in isolation.
As the best definition I consider: Unit is a functionality with well-defined boundaries (via API) which carries out a business requirement. This definition goes thru all the types of tests and should be always borne in mind by writing tests. Depending on the level of abstraction the unit can be a class, function, microservice, even the entire system.
This definition is tricky because it doesn't match the general picture we have about what Unit Testing is. Unfortunately, the name is so common that it's impossible to change it. Further in the text we will distinguish test unit (unit under test) from Unit Testing (testing on the lowest level of abstraction).
Another tricky term is isolation. Because in real system there are almost no truly isolated components - components work together as a system to provide a feature - so it makes no sense to try to isolate the components by stubbing or mocking them. Such tests have no big value. Consider following pseudo code:
public class GetProductFunction {
private static final DbClient db = DbClient.build();
public Response handle(Request request, Context context) {
DbResult result = db.query("ProductTable", "name, description", "id = ?", request.getProductId());
return result.found()
? new Response(200, result.getSingleItem())
: new Response(400);
} }
Building a database client is an expensive operation so we keep the instance as a static variable while the function container is warm.
Now we want to test this simple function in isolation. The problem is the variable db is private and final so if we want to stub it we propably end up with a code change like following:
public class GetProductFunction {
static DbClient db;
private static DbClient getDbClient() { return db == null ? db = DbClient.build() : db; }
public Response handle(Request request, Context context) {
DbResult result = getDbClient().query("ProductTable", "name, description", "id = ?", request.getProductId());
return result.found()
? new Response(200, result.getSingleItem())
: new Response(404);
} }
Changing the code for sake of a test sucks. There is a lot of problems with this code, but at least we can mock our resource now:
public class GetProductFunctionTest {
@Mock
private DbClient dbClient;
@Mock
private Request request;
@Mock
private Context context;
@Before
public void setup() { GetProductFunction.db = dbClient; /* mock the expensive resource */ }
@Test
public void testSuccess() {
when(request.getProductId()).thenReturn("test-id");
when(dbClient.found()).thenReturn(true);
when(dbClient.getSingleItem()).thenReturn(
Result.builder()
.put("name", "test-name")
.put("description", "test-description")
.build());
Response response = new GetProductFunction().handle(request, context);
assertEquals("test-name", response.get("name"));
assertEquals("test-description", response.get("description"));
}
@Test
public void testNotFound() {
when(request.getProductId()).thenReturn("test-id");
when(dbClient.found()).thenReturn(false);
when(dbClient.getSingleItem()).thenThrow(RuntimeException.class);
Response response = new GetProductFunction().handle(request, context);
assertEquals(404, response.getStatusCode());
} }
When we execute this test it's green - cool, great job! Wait, really? What do we actually test here? Apart from the trivial logic (found ⇒ 200, not found ⇒ 404) we test only the behavior we stubbed with the mocks. What's the value of testing mocks? - not a big one I guess...
As we saw in the previous example, besides a few exceptions it makes only sense to test components in the system. The unit of test isolation is the test. It means no other test must have any impact on execution or results - tests should run isolated from each other (possibly in parallel). The unit of test isolation is not a class, function or component.
We always test only the API (black-boxing), never the implementation (white-boxing)! There can be cases where implementation testing makes sense, or when we have to mock expensive resources in integration tests, but event in such situations coupling tests with the implementation should be used as little as possible.
Test Categories
Bases on the levels of abstraction of components under test we can distinguish several test categories and which APIs should be visible for the test:
| Test category | Component under test | API |
|---|---|---|
| Unit Testing | Function | Public methods / exported functions |
| Integration Testing | Microservice | Events (REST requests / messages) |
| End-to-End Testing | System | API Gateway |
What about testing of the (client) applications? Well, an application is actually an individual system so we can apply the entire testing model on it.
The test category determines how and where in the build pipeline the test is executed in:
A software delivery strategy cloud look like following scenario (an example):
- a commit triggers the pipeline
- the code source is fetched
- unit tests are being executed for each function (A, B, ... X)
- the code is being built for each function (A, B, ... X)
- the service is staged into a development environment, all resources are created
- integration tests are being executed for the entire service
- the component is deployed into a QA environment
- end-to-end testing pipeline is triggered or scheduled to run tests with a new version of the system
- application pipeline is triggered to be tested with a new version of the system
- the system is ready to be released into production
Unit Testing
Unit Tests operate on the lowest level. Unit Tests are parts of the codebase, having direct access to code they can test units in different layers without necessarily being deployed. As all the other tests, Unit Tests verify correct behavior only thru units API - interfaces or public functions - they never test the implementation.
Unit Tests verify business requirements. They are executed in the build phase, because it makes no sense to deploy code which doesn't match business rules.
Unit Tests should be written by developers in the same programming language as the function and be a part of the function code base.
As an example we have an AWS Lambda create-product* and the function createProduct() as the test unit; the function implements business rules saying the product price must be greater than zero and the product name must be included in the product description (probably for SEO reasons). Consider this Node.js code:
// create-product.js
const uuidv1 = require('uuid/v1')
module.exports = ({name, description, price}) => {
if (price <= 0) throw new Error('Price must be greater than zero.')
if (!description.includes(name)) throw new Error('Description must include name.')
return { id: uuidv1(), name, description, price}
}
// handler.js
const AWS = require('aws-sdk')
const dynamoDb = new AWS.DynamoDB.DocumentClient({apiVersion: '2012-08-10'})
const createProduct = require('./create-product')
exports.handler = async (event) => {
try {
const product = createProduct(event)
persistProduct(product)
return { statusCode: 201, body: product.id }
} catch (err) {
return { statusCode: 400, body: err.message }
}
}
async function persistProduct(product) {
const params = {
TableName: 'ProductTable',
Item: product
}
await dynamoDb.put(params).promise()
}
And a test for it:
describe('Unit test to create a product.', () => {
const testProduct = {
name: 'Product 123',
description: 'Product 123 Desc',
price: 123.4
}
const createProduct = require('../create-product')
it('Product should be created.', () => {
const product = createProduct(testProduct)
expect(product.id).toBeDefined()
})
it('Product must be greater than zero.', () => {
const product = { ...testProduct, price: 0 }
expect(() => createProduct(testProduct)).toThrow()
})
it('Description must include product name.', () => {
const product = { ...testProduct, description: 'junk' }
expect(() => createProduct(testProduct)).toThrow()
})
})
Integration Testing
Integration Testing in serverless systems operates on level of microservices. We test the whole use-case by verifying communication and interactions between functions and other resources.
Because Integration Tests are testing already deployed services there is no more the requirement to use the same programming language as the service is written in (we can all it "polyglot microservices"). But as well as Unit Tests the Integration Tests should be created and maintained by developers.
const AWS = require('aws-sdk')
const lambda = new AWS.Lambda({apiVersion: '2015-03-31'})
describe('Integration test to create and persist a new product.', () => {
const testProduct = {
name: 'Product 123',
description: 'Product Desc 123',
price: 123.4
}
it('Product should be available after created.', async () => {
// create a new product
const createParams = {
FunctionName: 'create-product',
Payload: JSON.stringify({
name: testProduct.name,
description: testProduct.description,
price: testProduct.price
})
}
const createResponse = await lambda.invoke(createParams).promise()
expect(createResponse).toBeDefined()
expect(createResponse.statusCode).toBe(201)
expect(createResponse.body).toBeDefined()
const productId = createResponse.body
// the product must be available now
const getParams = {
FunctionName: 'get-product',
Payload: JSON.stringify({
id: testProduct.id
})
}
const getResponse = await lambda.invoke(getParams).promise()
expect(getResponse).toBeDefined()
expect(getResponse.statusCode).toBe(200)
expect(getResponse.body).toBeDefined()
const foundProduct = JSON.parse(getResponse.body)
expect(foundProduct).toEqual(testProduct)
})
function sleep(ms) { /* ... */ }
})
End-to-End Testing
The end-to-end tests validate behavior of the entire system or its sub-systems and run apart from a feature (microservice's) build pipeline.
They can have a form of acceptance tests where whole paths are tested (1. search a product, 2. order the product, 3. create an invoice, 4. delivery etc.) or automated GUI tests which are testing the system from the client's point of view.
End-to-end tests could be generally written by someone else than by the developer. Especially GUI tests can be fully in hands of testers.
Of course, the manual testing belongs to this category as well.
In the previous integration-testing example both functions for saving and searching products were a part of one microservice - it means they lie in one build pipeline. If these functions were parts of different microservices the test would belong to the end-to-end testing category.
As an example let's consider a Catalog microservice - after creating a new product via create-product function a domain event like NEW_PRODUCT_CREATED should be fired. The event is a part of the internal API of the service and can be consumed by any other service registered to it. Typically an indexing service reacts to the event by extracting meta-data from the product and provides a full-search upon them. There should be an integration test validating this behavior.
End-to-end Testing doesn't access components directly but only via an API Gateway (e.g. REST).
public class CreateProductServiceTestE2E {
import ...
private final String SERVICE_API = System.getenv("SERVICE_API");
private final String PRODUCT_CREATE_ENDPOINT = SERVICE_API + "/catalog/product";
private final String SEARCH_ENDPOINT = SERVICE_API + "/search";
private HttpClient httpClient;
@Before
public void setup() { this.httpClient = createHttpClient(); }
@Test
public void testProductMetadataShouldIndexed() throws Exception {
Product product = new Product("Product 123", "Product Desc 123", 123.4);
// create a new product
HttpPost createRequest = httpClient.post(PRODUCT_CREATE_ENDPOINT);
createRequest.setHeader("Content-type", "application/json");
createRequest.setBody(product.toJson());
HttpResponse createResponse = httpClient.execute(createRequest);
assertEquals(201, createResponse.getStatusCode());
assertNotNull(createResponse.getBody());
String productId = createResponse.getBody();
Thread.sleep(1000); // waiting necessary due to eventual consistency
// the product must be indexed and available for searching
HttpPost searchRequest = httpClient.post(SEARCH_ENDPOINT);
searchRequest.setHeader("Content-type", "application/json");
searchRequest.setBody("{\"keyword\":\"" + product.getName() + "\"}");
HttpResponse searchResponse = httpClient.execute(searchRequest);
assertEquals(200, searchResponse.getStatusCode());
assertNotNull(createResponse.getBody());
Collection<String> foundProductIds = toList(createResponse.getBody());
assertTrue(foundProductIds.contains(productId));
}
}
Summary
In this article I wanted to recap my experience with functional testing of serverless systems and simplify the levels of tests. I reviewed terms Unit-, Integration- and End-to-end Testing to show what is under test and where such tests lie in the development process.
Despite the fact that I focused only on functional testing doesn't mean that testing non-functional requirements like performance, security, scalability, etc. are not important...:-)
Happy testing!
* I use verbs to name functions because it sounds natural to me. A function is in fact an action.