Quick Reference

Glossary

Unit test

A test that covers a single functions or method (a “unit”)

Integration test

A test that checks if “units” work together as intended

Smoke Test

Check that the whole application or script runs without errors in the simplest scenario possible. If this fails, no point in testing other things, usually

Regression

A loss of functionality, typically due to a bug

End-to-end test

Test the whole thing running, checking the output (For example, running on sample data and checking that the output is the expected one. See also Regression test)

Regression test

Check that results and behaviour are what they are supposed to be. Typically written once a regression is detected. Other names for the same kind of test:

  • Characterization Tests

  • Acceptance Tests

  • Golden-Master Tests

Characterization Test

Automated test that is written on existing (legacy) code, (assuming that version of the code is correct) and added to a test suite to make further work/changes easier

Test-first development

The practice of writing automated tests before writing the code that makes the tests pass

TDD

Acronym for Test-driven development

Test-driven development

A special case of Test-First development where the workflow is:

  • Make a list of specifications

  • Then, for each specification:

    • Write a test, run it and verify that the test fails

    • Write the minimum amount of code to make the test pass

    • Refactor and improve the code

Continuous integration

The practice of merging in the main branch frequently without having long-lived branches. This typically requires automating part of the workflow, especially testing, and GitHub/GitLab et similia have support for that via Actions/CI-CD respectively.

Code coverage

Metric representing the fraction of code base executed during the test suite.
Note: this is only an upper bound to the fraction of code base that is anyhow tested. And it is perfectly possible to write tests that are completely useless but increase code coverage.

Testing framework

Framework that runs tests for you. See the following for some examples.

Linter

A program that can check your code for typical mistakes or for risky practices, and reports them to you.

Fixture

A resource that needs to be set up before a test case can run and needs to be torn down after the test case (or a whole test suite) has run.

Property testing

Test that a property of the code holds for a whole class of inputs. Typically done by automatically generating test cases according to a strategy. Tends to very time-consuming compared to unit testing.

Test Registration

The act of marking a test for execution in a main testing program. Test frameworks allow to do this automatically at test definition so that it does not have to be manually invoked in the “main” script/function, with the risk of forgetting it.

Test Pyramid

The general approach for balancing test types in a test suite. The slower a test type is, the fewere tests of that type should be in the test suite.

Search engines can show many representations of this.

Available tools

Unit test frameworks

A test framework makes it easy to run tests across large amounts of code automatically. They provide more control than one single script which does some tests.

pytest

  • Python

  • https://docs.pytest.org/

  • Installable via Conda or pip.

  • Easy to use: Prefix a function with test_ and the test runner will execute it. No need to subclass anything.

def get_word_lengths(s):
    """
    Returns a list of integers representing
    the word lengths in string s.
    """
    return [len(w) for w in s.split()]

def test_get_word_lengths():
    text = "Three tomatoes are walking down the street"
    assert get_word_lengths(text) == [5, 8, 3, 7, 4, 3, 6]

testthat

  • R

  • https://github.com/r-lib/testthat

  • Easily installed from CRAN with install.packages("testthat"), or from GitHub with devtools::install_github("r-lib/testthat")

  • Use in package development with usethis::use_testthat()

  • Add a new test file with usethis::use_test("test-name"), e.g.:

    # tests/testthat/test_example.R
# file added by running `usethis::use_test("example")`

context("Arithmetics")
library("mypackage")

test_that("square root function works", {
  expect_equal(my_sqrt(4), 2)
  expect_warning(my_sqrt(-4))
})

    Tests consist of one or more expectations, and multiple tests can be grouped together in one test file. Test files are put in the directory tests/testthat/, and their file names are prefixed with test_.

  • Run all tests in package with devtools::test() (if you use RStudio, press Ctrl+Shift+T):

    > devtools::test()
Loading mypackage
Testing mypackage
✔ |  OK F W S | Context
✔ |   2       | Arithmetics

══ Results ═════════════════════════════════════════════════════════════════════
OK:       2
Failed:   0
Warnings: 0
Skipped:  0

More information in the Testing chapter of the book R Packages by Hadley Wickham.

Test

  • Julia

  • Part of the standard library

  • Provides simple unit testing functionality with @test and @test_throws macros:

julia> using Test

julia> @test [1, 2] + [2, 1] == [3, 3]
Test Passed

# approximate comparisons:
julia> @test π ≈ 3.14 atol=0.01
Test Passed

# Tests that an expression throws exception:
julia> @test_throws BoundsError [1, 2, 3][4]
Test Passed
      Thrown: BoundsError

julia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]
Test Passed
      Thrown: DimensionMismatch

  • Grouping related tests with the @testset macro:

using Test

function get_word_lengths(s::String)
    return [length(w) for w in split(s)]
end

@testset "Testing get_word_length()" begin
    text = "Three tomatoes are walking down the street"
    @test get_word_lengths(text) == [5, 8, 3, 7, 4, 3, 6]
    number = 123
    @test_throws MethodError get_word_lengths(number)
end

Catch2

#include <catch2/catch.hpp>

#include "example.h"

using namespace Catch::literals;

TEST_CASE("Use the example library to add numbers", "[add]") {
  auto res = add_numbers(1.0, 2.0);
  REQUIRE(res == 3.0_a);
}

Google Test

  • C++

  • Documentation

  • Widely used

  • Very rich in functionality

  • Well-integrated with CMake

#include <gtest/gtest.h>

#include "example.h"

TEST(example, add) {
  double res;
  res = add_numbers(1.0, 2.0);
  ASSERT_NEAR(res, 3.0, 1.0e-11);
}

Boost.Test

  • C++

  • Documentation

  • Very rich in functionality

  • Header-only use possible

#include <boost/test/unit_test.hpp>

#include "example.h"

BOOST_AUTO_TEST_CASE( add )
{
  auto res = add_numbers(1.0, 2.0);
  BOOST_TEST(res == 3.0);
}

pFUnit

@test
subroutine test_add_numbers()

   use hello
   use pfunit_mod

   implicit none

   real(8) :: res

   call add_numbers(1.0d0, 2.0d0, res)
   @assertEqual(res, 3.0d0)

end subroutine

To test the factorial and fizzbuzz functions from the test-design exercises, use this CMakeLists.txt file:

cmake_minimum_required(VERSION 3.12)

project (PFUNIT_DEMO_CR
  VERSION 1.0.0
  LANGUAGES Fortran)

find_package(PFUNIT REQUIRED)
enable_testing()

# system under test
add_library (sut
  factorial.f90
  fizzbuzz.f90
  )

target_include_directories(sut PUBLIC ${CMAKE_CURRENT_BINARY_DIR})

# tests
set (test_srcs test_factorial.pf test_fizzbuzz.pf)
add_pfunit_ctest (my_tests
  TEST_SOURCES ${test_srcs}
  LINK_LIBRARIES sut
  )

You can then compile using this script:

#!/bin/bash -f

if [[ -d build ]]
then
    rm -rf build
fi

mkdir -p build
cd build
cmake .. -DCMAKE_PREFIX_PATH=$PFUNIT_DIR
make
./my_tests

# or
# ctest --verbose

Services to deploy testing and coverage

Each of these are web services to handle testing, free for open source projects.

Good resources

Keypoints

  • Testing is a basic requirement of any possible language

  • There are various tools for any language you may use

  • There are free web services for open source