Automated testing in Umbraco
Making testing easier to write and maintain
While migrating Umbraco to .NET Core, all of the automated tests also need to be migrated. First of all, the old C# test suite is running .NET Framework 4.7.2. We started by creating two new empty test suites, one for Unit Tests and one for Integration Tests. Both run .NET Core 3.1.
Our automated tests are extremely important while migrating the solution in small batches because we run all the automated tests for each Pull Request, so we have a better idea that the changes actually work as expected. That being said, it is important to emphasize that just because all automated tests pass successfully does not mean that no errors have been introduced. We don’t have all scenarios covered by tests.
The purpose of different test strategies
The main purpose of all tests is to prove a single scenario is working as expected. We can separate the test strategy into two categories based on the knowledge required by the tester:
- Black-box tests
are test methods in which the internal structure/ design/ implementation of the item being tested is not known.
- White-box tests
are test methods in which the internal structure/ design/ implementation of the item being tested is known.
Let’s take some examples, to make this more clear. Consider the following code, and imagine we want to test the CalculatorWrapper class:
The following code examples illustrate the different types of tests:
The difference is that in the black-box test, we don’t care about how the result is generated, as long as it is correct. These types of tests are very stable when the codebase is refactored to improve maintainability and, as in our case, update the underlying frameworks.
On the other hand, the white-box test can be written before the actual implementation of the Calculator class. This is a massive advantage when working on big projects. This leads to the different types of automated tests we have in Umbraco.
Unit tests
Unit tests are the test of small logical units, often a single method. But in theory, it can be a larger unit. In my opinion, the most important part is that the test can be executed in at most a few milliseconds.
We isolate the unit by replacing all dependencies with mocks/fakes. This is easy when all dependencies are injected into the class or method, but can be very hard if not. That’s also why we have updated a lot of classes to inject the dependencies while migrating to .NET Core.
We have unit tests written in both C#, for the backend code, and Javascript for the client code.
In the backend, we use the test framework NUnit. The examples earlier would be considered as Unit Tests using NUnit. In theory, we could replace our implementation of the ICalculator interface and use a web service to do the actual calculation. In that case, the black-box test would be an integration test, and the white-box test still a unit test.
In Javascript, we use the Jasmine test framework and Karma test runner.
The following example shows a test of a truncate filter in Javascript:
This example has three test cases with different input and expected output.
Some of the most obvious scenarios for unit testing are generally input validation and edge cases, but of course unit tests can always be used, and their execution speed doesn't make the test suite noticeably slower.
Integration tests
Even though Unit Tests are extremely important to verify the code, they can’t do it alone. There are countless examples from the real world where small components (Units) work as expected, but when used in integration with other components the outcome is not as expected.
Consider the following Meme:
Both the door and the lock work as expected when tested in isolation. But when tested in integration with each other, the door can still not be locked.
The same can easily happen in software. That’s why integration tests are also very important.
Most often we test the integration between our services and the database, web-services or the filesystem when we write Integration tests. So instead of mocking repositories to return the expected results, we need to seed the database with the expected data before we execute our actual integration test.
In Umbraco, we have set up our integration test suite such that a new database can be provided for each test. This is a perfect starting point if you are going to test a Service and down to the database. Let’s take a look:
This test follows the classic Arrange-Act-Assert pattern. In this case, the act part includes the database seeding. Note that by using the UmbracoIntegrationTest base class, we can get services we need using the GetRequiredService method.
Sometimes we want to test from a controller action and down to the database. In this case, we use the built-in concept of a test server. All you need to do is to use the base class UmbracoTestServerTestBase. Let’s take an example:
In this example you have to note two things:
- The PrepareUrl to get the URL of an Action and ensure all services use the URL information when requested.
- The Client which is a normal HttpClient, but the base URL points to the test server that is set up for each test.
Note that you can still use GetRequiredService to get the services required to seed data.
Some of the most classic scenarios for integration testing are in general the Happy path of the method and some of the specific MVC concepts like Action filters.
Keep in mind that integration tests require a lot of setup before the test executes. So, if input validation tests are created using the UmbracoIntegrationTest base class, these will still succeed, but the execution time will be many times longer compared to a unit test.
Acceptance tests
The last type of automated test we have in Umbraco is Acceptance tests. These are implemented using the Cypress framework.
The acceptance tests use the UI to communicate with the backend. So these kinds of tests ensure that the client code and server code integrate as expected. While there are countless benefits to these kinds of tests, there still are some challenges. As we can’t spin up a new Umbraco solution for each test, we need to ensure that the solution is in a known state before the test starts. We also need to clean up when the test is done.
Our acceptance tests are written in TypeScript/Javascript and use some of the common test frameworks in these languages. Let’s take a look at an example:
This is a short test that verifies that the Backoffice login is working. It reads the username and password from a configuration file, types these into the form, and clicks the login button. Finally, it asserts the URL is changed to the expected URL after login and ensures the login form is not shown anymore.
Another example which creates data and needs to be cleaned up is the following:
In this example, we test that the delete template functionality is working as expected.
We start by ensuring there are no templates with the name we are about to create. Then we create and save a dummy template.
We go to the settings section, verify the settings tree is visible and right-click on the template with the given name. Click Delete and OK. Then we assert the item is deleted from the menu.
Finally, we clean up, using APIs, just in case the functionality is not working as expected.
Acceptance tests are useful for testing use cases exposed in the UI. The biggest benefit of acceptance tests regarding the .NET Core migration is that the tests can be reused, without us having to modify the tests.
Debuggability
The only reason we keep tests after we have verified they succeed is that we fear the functionality will fail in the future. Therefore it is also important that we have prepared the test for the developer that has to debug the failing functionality in the future.
I have seen plenty of tests failing with an error message without any context, like this example:
Expected: True
But was: False
This is an error message returned from a Assert like the following:
Assert.IsTrue(new []{"A", "B", "C"}.Contains("D"));
Instead, we could change the assert into
CollectionAssert.Contains(new []{"A", "B", "C"}, "D");
Meaning we would have a way better error message, like this:
Expected: some item equal to "D"
But was: < "A", "B", "C" >
This is a classic example of how to be friendly to yourself or other developers on the project. In general, always look into CollectionAssert if you are asserting anything on a collection.
You can also provide a custom error message to the assert, which can be very helpful.
Sometimes your tests have multiple asserts. In that case, we only see the error message of the first failing assert.
To show all the error messages, we need to wrap all the assets into an Assert.Multiple. By making these small changes, you make it way easier to debug a test in the future.
Maintainability
Smart developers have always said the code quality of the test suite has to be the same as the production code. In reality, I have never seen this.
One of the most unused principles of test suites seems to be DRY. I have seen countless tests where the entire arrange part was copied and most of the copied arrange part is not even necessary for the new test.
One example from our own codebase is here:
The best way to avoid this is to make it easy to create the data needed for a test. I’m a big fan of the builder pattern to make it easy to create test data.
While the builder pattern is implemented in frameworks like AutoFixture, I still prefer to explicitly implement them to have 100% control. Consider the following example:
new UserBuilder().Build();
This simple line returns a valid User. All required properties are populated. The cool thing is that we easily can customise the output, like in the following example:
new UserBuilder()
.WithUsername(“Admin”)
.Build();
This example returns a valid user, with the “Admin” as Username.
Builders can also be nested, e.g. the User object has a nested UserGroup. We handle this in the builders in the following way.
new UserBuilder()
.WithUsername(“Admin”)
.AddUserGroup()
.WithAlias("writer")
.Done()
.Build();
This example returns a valid user, with the “Admin” as Username and a single user group with alias “writer”. If we serialize the output of that builder, it is clear how much dummy data we did not have to think about, even that some of it is required to be allowed to work with the User class:
Overall we are attempting to improve the quality of our test suites, even if that is not strictly required to make the tests run in .NET Core. We have separated the integration tests from the unit tests to allow us to execute the majority of tests in a matter of seconds instead of about 15 minutes.
We introduced acceptance tests into Umbraco 8 because these tests can be reused for both Umbraco 8 and the .NET Core version. Now we just need to migrate the renaming tests and write a lot of new acceptance tests. This is also an area where community contributions are very welcome, so feel free to reach out 😊