Shades of Tests
Practical examples of software testing in Java using Spring Boot Test without mocking frameworks.
Disclaimer: For purposes of this text, when I’m talking about testing I mean functional testing. Albeit non-functional testing is an important part of software development I will ignore it here entirely.
There are many shades of testing in software development. It’s handy for every developer to know the differences and trade-offs to make the testing as effective as possible.
I use the word “effective” on purpose as I see a lot of effort invested in making testing as efficient as possible, but unfortunately in the wrong kind of testing.
Features-rich tools are nice to write and execute end-to-end tests efficiently, but results will never be as quick, cheap, applicable and stable as simple but numerous unit tests.
It’s like trying to optimize a bubble-sort algorithm: you can make it quite efficient, but it still will never beat the merge-sort.
Understanding different kinds of tests and where to use what is crucial for effectivity.
The Test Pyramid
When talking about testing enterprise software, the (in)famous Test Pyramid always comes up.
Developers find the Test Pyramid often annoying for two main reasons:
- it doesn’t fit the product nature, or
- it’s not completely understood.
The former could be really justified, especially for GUI software or projects that focus particularly on integration. That’s why I have to emphasize I will talk about enterprise software development exclusively for the rest of this text.
The latter reason for the dislike is the problem I’d like to address.
Simple Test Pyramid
The common understanding of the Test Pyramid is its volume aspect: the bigger the volume the bigger the amount of tests of that kind. Simply put, the test base should contain a lot of unit tests, some integration tests, and just a few UI tests.
However, the Pyramid has more to offer. With the increasing volume increase speed and costs as well. While unit tests are quick and cheap, UI tests tend to be very slow and expensive.
Test Pyramid with speed and cost axis
The reason is in the complexity of the tests. Integration tests are obviously more complicated than unit tests and so on. More components are involved, more communication must be worked out, the computer(s) must execute more instruction. Tests are more difficult (expensive) to develop, set up, execute.
These additional aspects explain the economic reasons behind the different recommended amounts. And what’s more, there is another characteristic we can put to the axis: applicability.
With increasing volume of the Pyramid the applicability of tests declines. It’s obvious that unit tests of the core domain will always apply, but there could be several UIs accessing its functionality. Components can be integrated into multiple different systems as well.
With the absence of unit tests, integration tests would need to be duplicated for every system and UI tests for every client. This is expensive, slow, and tedious. Besides, the core domain is usually the most stable piece of software while its clients tend to be pretty volatile. This leads us to another aspect: stability.
Tests that frequently require changes and adjustments are often abandoned or ignored. Stability of tests goes down with decreasing volume of the Pyramid.
Last but not least, tests on the top of the Pyramid are often flaky. Flaky tests are especially annoying when the test suite is slow.
Test Pyramid characteristics
Kinds of Tests
The terminology is vague and ambiguous and there are no official definitions of different types of tests. The terms are overloaded and can obtain various meanings among different people.
I’m not aspiring to standardize the terminology, you are free to use any name you like. However, to discuss the concept it is important to unify the name, at least for purposes of this text.
We will refine the Test Pyramid into the following levels:
Extended Test Pyramid
(Atom) Unit Tests
This definitely the most unfortunate name of all. The problem is, a unit doesn’t necessarily be a small piece of software with a single responsibility and a well-defined API for a business capability, as it should in the context of unit testing. A unit means generally any piece of software, in an extreme case the entire system.
Albeit it’d be really handy to have a more descriptive name for the unit test, unfortunately I’m not aware of any. We can try it with things like element tests or atom tests to emphasize the atomicity of the unit under test, but the term unit tests is strongly adopted and ubiquitous and is here to stay with us forever.
Atom has no dependencies
How does a unit test look like anyway? We don’t really need any special magic here:
class PromoProductTest {
@Test
void promo_product_is_not_deliverable() {
Product product = new PromoProduct(
"Test product", 123);
assertThat(product.deliverable()).isFalse();
}
}
No dependencies, tested business capabilities against the API, this is a nice unit test.
Here is the beauty about unit testing: unit tests require code to be testable.
Testable code must be simple, cohesive and loosely coupled. Easy to say, hard to do. That’s usually the reason a codebase has so few unit tests - they are hard to write because the code is not testable.
Tests are the very first client of the code and help you to understand the API to be easy to use and understand.
Another benefit of good tests is their documentary function. Tests show the proper usage of code to serve as an exhaustive user reference.
I often refer to testing as to the “king’s discipline” of software development. Only good developers can do it right.
True (atom) unit tests require the domain model to be pure. This is not always easy or even possible to achieve.
Component Tests
When purity is not feasible, test doubles come to help. External dependencies such as databases or filesystems can be stubbed for testing via alternative implementations of resource adapters.
It’s important to focus on the business functionality. With test-doubles you can easily find yourself writing tests that check only stubbed calls or integration with doubles. In this case, it’s time to move on to component testing.
Component tests lie somewhere between unit and integration tests. They are no pure unit tests as more than one “atom” is involved.
However, component tests are no pure integration tests either, as the unit is not completely and correctly integrated - stubbing is often used, only directly needed components are integrated, not the whole system.
Component’s dependencies are stubbed
Spring Boot offers so-called "slicing" where only needed pieces of the system are included in the test:
@DataJpaTest
@Import(JpaProductConfig.class)
class PromoteTest {
@Autowired
private Promote promote;
@Test
void product_is_promoted() {
var product = promote.product(new Promote.ToPromote(
"Test product", 123));
assertAll(() -> assertThat(product.id()).isNotNull(),
() -> assertThat(product.deliverable()).isFalse());
}
}
Integration Tests
Integration tests check the component fully integrated in the system. The system is seen as a black box accessible only via its public APIs.
The system is usually bootstrapped from within the test, which makes it possible to run the system with special configuration. For example, a test database with a testing dataset could be used.
Also, the component under the test is typically deployed into the core system without any other components.
Component is integrated in a system slice
Spring Boot brings the @SpringBootTest annotation to support integration testing:
@SpringBootTest(classes = Application.class,
webEnvironment = RANDOM_PORT)
class ProductsTestIT {
@LocalServerPort
private int port;
@Test
void product_is_promoted() {
with()
.port(port)
.basePath("/products/promoted")
.contentType(ContentType.JSON)
.body("{\"name\":\"Test\","
+ "\"price\":123}")
.post().
then()
.statusCode(201)
.body("id", is(notNullValue()))
.body("name", equalTo("Test"))
.body("price", equalTo(123));
}
}
End-to-End Tests
End-to-end tests, also known as system tests, runs typically in three phases:
- preparation, system startup;
- testing, tests execution;
- tear down, system shutdown, cleanup.
As the system is started only once, there is no room for adjustments based on special tests’ needs. The system runs as a separate process, deployed in an environment as similar to production as possible.
Component is integrated in the whole system
Maven Failsafe Plugin address this kind of testing, as well as Testcontainers:
@Testcontainers
class ProductsTestE2E {
@Container
public GenericContainer app = new GenericContainer(
new ImageFromDockerfile()
.withFileFromPath("app.jar",
Path.of("build/libs/app.jar"))
.withDockerfileFromBuilder(b -> b
.from("openjdk:11-jre-slim")
.copy("app.jar", "/app.jar")
.entryPoint("java", "-jar", "/app.jar")
.build()))
.withExposedPorts(8080);
private int port;
@BeforeEach
public void setUp() {
port = app.getFirstMappedPort();
}
@Test
void product_is_promoted() {
with()
.port(port)
.basePath("/products/promoted")
.contentType(ContentType.JSON)
.body("{\"name\":\"Test\","
+ "\"price\":123}")
.post().
then()
.statusCode(201)
.body("id", is(notNullValue()))
.body("name", equalTo("Test"))
.body("price", equalTo(123));
}
}
UI Tests
In the context of this text, UI testing is different from testing a UI.
UIs themselves should be unit-tested in isolation against a stubbed backend, but this is no full replacement for UI testing a UI integrated in the system.
UI tests in enterprise software development are end-to-end tests with actions triggered from a GUI (web or desktop) instead of Restful or similar APIs.
Component is accessed via UI
UI tests tend to be very slow, unstable and flaky. Unfortunately, it is still not unusual to have a test base containing UI tests mostly or even only.
The test base should contain UI tests for elementary user interactions with the UI. Especially to check edge conditions of user input, validation and error handling towards the user.
The amount of UI tests should be kept on a small maintainable amount, everything must be automated.
There are plenty of tools for automation of UI testing such as Selenium:
class ProductsTestUI {
private WebDriver webDriver;
@BeforeEach
void setUp() {
webDriver = new HtmlUnitDriver();
webDriver.get("http://localhost:8080/ui");
}
@Test
void product_is_promoted() {
var form = webDriver.findElement(By.id("form-promote"));
form.findElement(By.className("name")).sendKeys("Test");
form.findElement(By.className("price")).sendKeys("123");
form.findElement(By.className("submit")).click();
var list = webDriver.findElement(By.id("products-promo"))
.findElements(By.className("product"));
assertAll(
() -> assertThat(list).hasSize(1),
() -> assertThat(list.get(0).findElement(
By.className("id")).getText()).isNotNull(),
() -> assertThat(list.get(0).findElement(
By.className("name")).getText()).isEqualTo("Test"),
() -> assertThat(list.get(0).findElement(
By.className("price")).getText()).isEqualTo("123")
);
}
}
UX Tests
Albeit there is no excuse to have an automatic test for everything, one kind of testing does require manual labor.
Look’n’feel and user experience (UX) in general cannot be evaluated by any machine, a human is required here.
User experience matters
Source Code
The source code for discussed examples can be found on my GitHub.
Happy testing!