Motivation

Objectives

Appreciate the importance of testing software
Understand various benefits of testing

Most scientists nowadays depend on software for research.
What can go wrong when research software has bugs? Look no further:

How can we avoid problems like these?

What are typical problems that automated tests can address?

Have you ever had any of these problems?

You change B and C, and suddenly A doesn’t work anymore. Time wasted trying to figure out what changed.
There was some simple problem, systematically testing could have found it. But testing manually takes too much time, so nobody has ever done it with the appropriate care.
You get someone else’s code. You really need to change it but are afraid to touch it because who knows what might break. Plot twist: it’s your own code!
You implement features to someone else’s code and want to merge it, but they are not sure your changes haven’t broken anything and it’s time consuming to test that.

People have learned that some automatic way to check problems makes software development much easier. This lesson will talk about the places it’s useful for research code, and how easy it can be.

Untested software can be compared to uncalibrated measurement devices

Before relying on a new experimental device, an experimental scientist always establishes its accuracy. A new detector is calibrated when the scientist observes its responses to known input signals. The results of this calibration are compared against the expected response.

—From Testing and Continuous Integration with Python, created by K. Huff

With testing, simulations and analysis using software can be held to the same standards as experimental measurement devices!

What can tests help you do?

Problems, Solution and who is affected?
Problem	Solution	Who is affected?
Breaking old functionality when adding new features	End-to-End tests	Developers
Verify installation	Smoke tests	Users
Make small incremental changes, (e.g., improving readability, names)	Unit tests	Developers
Make architectural changes, (e.g., shifting code between classes, modules and functions)	Integration tests End-to-End tests	Developers
Change things with confidence that nothing is breaking	All tests	Developers
Documentation out of date including code examples	Executable notebooks and nbval, End-to-End tests	Users

Very few people are proud of the code they write the first time they write it.

Often, they’d like to improve it.

But code without automated tests cannot be improved as easily as code with automated tests.

Moreover, code that is easy to test is probably easier to maintain, since it needs to be more modular and have better separation of concerns.

The Modular code development lesson demonstrates this.

Testing in a nutshell

There are many forms of testing.

One can write test programs and run them (a form of end-to-end testing):

$ python3 run-test.py

running: sample_data/set1.csv --output=tests/set1.txt
CORRECT

In the most basic form of a software test, the observed result is compared with expected result (an “oracle”) in order to establish correctness.
Here are some examples of this testing pattern in different programming languages (in this case, unit tests):

def fahrenheit_to_celsius(temp_f):
    """Converts temperature in Fahrenheit
    to Celsius.
    """
    temp_c = (temp_f - 32.0) * (5.0/9.0)
    return temp_c

# This is the test function: `assert` raises an error if something
# is wrong.
def test_fahrenheit_to_celsius():
    temp_c = fahrenheit_to_celsius(temp_f=100.0)
    expected_result = 37.777777
    assert abs(temp_c - expected_result) < 1.0e-6
    

#include <cmath>  // std::abs
#include <cstdlib>
#include <iostream>

using namespace std;

/* Converts temperature in Fahrenheit to Celsius. */
double fahrenheit_to_celsius(double temp_f) {
  auto temp_c = (temp_f - 32.0) * (5.0 / 9.0);
  return temp_c;
}

/* This is the test function: `throws` raises an error if something is wrong. */
void test_fahrenheit_to_celsius() {
  auto temp_c = fahrenheit_to_celsius(100.0);
  auto expected_result = 37.777777;
  try {
    if (abs(temp_c - expected_result) > 1.0e-6) throw "Error";
  } catch (char const* err) {
    cout << err;
  }
}

int main() {
  cout << fahrenheit_to_celsius(20);
  test_fahrenheit_to_celsius();
  return EXIT_SUCCESS;
}

# Converts temperature in Fahrenheit to Celsius.
fahrenheit_to_celsius <- function(temp_f)
{
  temp_c <- (temp_f - 32.0) * (5.0/9.0)
  temp_c
}

# This is the test function: `assertive::is_true` raises an error if something
# is wrong.
test_fahrenheit_to_celsius <- function()
{
  temp_c <- fahrenheit_to_celsius(temp_f = 100.0)
  expected_result <- 37.777777
  assertive::is_true(abs(temp_c - expected_result) < 1.0e-6)
}

using Test

"""
    fahrenheit_to_celsius(temp_f::Float)

Converts temperature in Fahrenheit to Celsius.
"""
function fahrenheit_to_celsius(temp_f)
    temp_c = (temp_f - 32.0) * (5.0/9.0)
    return temp_c
end


# This is the test section
@testset "Test fahrenheit_to_celsius" begin
    temp_c = fahrenheit_to_celsius(100.0)
    expected_result = 37.777777
    @test abs(temp_c - expected_result) < 1.0e-6
end

program temperature_conversion

implicit none
call test_fahrenheit_to_celsius()

contains

function fahrenheit_to_celsius(temp_f) result(temp_c)
   implicit none
   real temp_f
   real temp_c
   temp_c = (temp_f - 32.0) * (5.0/9.0)
end function fahrenheit_to_celsius

subroutine test_fahrenheit_to_celsius()
   implicit none
   real temp_c
   real expected_result
   temp_c = fahrenheit_to_celsius(100.0)
   expected_result = 37.777777
   if( abs(temp_c - expected_result) > 1.0e-6) then
      write(*,*) 'Error'
   else
      write(*,*) 'Pass'
   end if
end subroutine test_fahrenheit_to_celsius

end program temperature_conversion

Discussion: When is it OK not to add automated tests?

Discussion: When is it OK not to add automated tests?

Vote in the notes and we’ll discuss soon. It is always a balance: there is no “always”/”never”.

Jupyter or R Markdown notebook which produces a plot and you know by looking at the plot whether it worked?
A short, “obviously correct” Python or R script which you never intend to reuse?
A simple short, “obviously correct” shell script?
Can you give other examples?

Solution

The role of automated tests is to save time when making changes to code.

In this case you just “test manually” the notebook by running it. Automated tests might not save you time. But if some non trivial functions are added, you might want to have automated unit tests for these separately.
Writing automated tests for “throwaway code” can be a waste of time. But if you get back to it, then you should think about writing automated tests for it.
“Manual test” can be sufficient. In case of changes, checking the script with a linter like Shellcheck might still be useful!

What should you do?

If code is interactive-only (Jupyter Notebook), it’s usually hard to test.
- But also hard to run: the next lesson will discuss!
At least end-to-end is often easy to add.
Add tests of tricky functions.
- If you’d have to run it over and over to test while writing, why not make it a property test?
It’s easy to have Gitlab/Github run the tests.
- It’s nice to push without thinking, and the system tells you when it’s broken.
Learning how to test well make the rest of your code better, too.