Testing policy

This section of the document outlines The Sensible Code Company’s (SCC) approach to testing Cantabular both throughout its development and for its release.

It covers the role and objectives of testing as we see it, the specific techniques we employ on a day-to-day basis, and our approach to creating and testing a new release of the software.

Introduction

SCC views testing as an integral part of the software development process: we spend as much time writing automated tests as we do creating new features.

We try to automate our tests as much as possible and always try to move in the direction of more testing, and more test automation, not less.

We test with multiple objectives in mind:

  • Code quality: new code is readable, well-formatted and free of common mistakes.

  • Correctness of outputs: data outputs produced by the software are exactly correct.

  • Identify bugs and regressions: new code must not cause errors in existing features and the process of writing tests for new features helps identify bugs in new code.

  • Consistent performance: the performance of software components should be maintained or improve over time.

  • Cross-browser compatibility of user interfaces: user interfaces should be functional and look as good as intended in a range of different browsers and devices.

Testing techniques

Our test suite is run on every change to the software, not just at release time. Tests are run on GitHub Actions and a pull request in GitHub containing a new feature has to pass the tests before it can be merged into the main branch.

Individual developers working on changes to the software run some or all of the test suite multiple times per day.

New features must be accompanied by appropriate tests, and this is enforced in mandatory code reviews.

Our tests are at many levels and the suite currently takes about 16 minutes to run on GitHub Actions. They cover:

  • Code linting and static analysis to enforce code style and formatting, and to identify common mistakes such as typos, bad use of APIs, weak constructs and concurrency issues.

  • Automated checks for security vulnerabilities in production code dependencies. When dependencies are forked then a test only dependency is created on the upstream repository so that the latter is still included in the security vulnerability check.

  • Automated checks for updates in Go dependencies that automatically create pull requests to update them.

  • Unit tests of Go code packages, code fragments and TypeScript functions.

  • OpenAPI tests that generate client code from the server API definition and then test it with a built server and dataset.

  • Integration tests within Go that construct artefacts (e.g. datasets, codebooks) in memory and test them server side.

  • Integration tests that construct in-memory servers with in-memory datasets and test APIs across HTTP (all of metadata API, extended API, server API).

  • Integration tests that run built executables in order to sanity test their overall function: building a dataset, loading it into the other software components, querying the software and inspecting responses and logging outputs.

  • Integration tests that run built executables and check query outputs for a batch of queries against expected results to ensure correctness.

  • User interface tests that run built executables with sample configuration and use a test framework (TestCafé) to drive a headless browser and click on UI elements and check rendered web pages function as expected.

During development, we also use BrowserStack to perform manual testing of new user interface features or layout changes in multiple browsers and devices.

For our public user interface, we adhere to the UK Government Digital Service’s recommended browsers to test in, ensuring compliance across desktop and mobile.

For our administration user interface, we do not attempt to support mobile devices as it is intended for use by expert users in a professional environment.

Release process

A new release of Cantabular is built automatically in GitHub Actions in a clean Ubuntu image. All of the automated tests as described above are run against it as part of that process.

In addition to those automated tests, we also:

  • Run an additional large set of queries based on data that cannot be in source control for confidentiality reasons to ensure correctness of output, proper operation of disclosure rules and to provide an indicator of query performance.

  • Where a release includes changes to the user interface, we run an automated penetration test against it using BurpSuite to ensure no new bugs or regressions in its security profile.