List of exercises

Full list

This is a list of all exercises and solutions in this lesson, mainly as a reference for helpers and instructors. This list is automatically generated from all of the other pages in the lesson. Any single teaching event will probably cover only a subset of these, depending on their interests.

In-code documentation

In in-code-documentation.md:

In-code-1: Comments

Let’s take a look at two example comments (comments in Python start with #):

Comment A

# now we check if temperature is below -50
if temperature < -50:
    print("ERROR: temperature is too low")

Comment B

# we regard temperatures below -50 degrees as measurement errors
if temperature < -50:
    print("ERROR: temperature is too low")

Which of these comments is more useful? Can you explain why?

Writing good README files

In writing-readme-files.md:

Exercise README-1: Have fun testing some README features you may not have heard about

  • Test the effect of adding the following to your GitHub README (read more):

    > [!NOTE]
> Highlights information that users should take into account, even when skimming.

> [!IMPORTANT]
> Crucial information necessary for users to succeed.

> [!WARNING]
> Critical content demanding immediate user attention due to potential risks.

  • For more detailed descriptions which you don’t want to show by default you might find this useful (please try it out):

    <details>
<summary>
Short summary
</summary>

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
</details>

  • Would you like to add a badge like this one: please replace with alt text?

    Badge that links to a website (see also https://shields.io/):

    [![please replace with alt text](https://img.shields.io/badge/anytext-youlike-blue)](https://example.org)

    Badge without link:

    ![please replace with alt text](https://img.shields.io/badge/anytext-youlike-blue)

  • Know about other tips and tricks? Please share them (send a pull request to this lesson).

In writing-readme-files.md:

Exercise README-2: Draft or improve a README for one of your recent projects

Try to draft a brief README or review a README which you have written for one of your projects.

  • You can do that either by screensharing and discussing or working individually.

  • Use the checklist which we have discussed earlier.

  • Think about the user (which can be a future you) of your project, what does this user need to know to use or contribute to the project? And how do you make your project attractive to use or contribute to?

  • (Optional): Try the https://hemingwayapp.com/ to analyse your README file and make your writing bold and clear.

  • Please note observations and recommendations in the collaborative notes.

In writing-readme-files.md:

Exercise README-3: Review and discuss a README of a project that you have used

In this exercise we will review and discuss a README of a project which you have used. You can also review a library which is popular in your domain of research and discuss their README.

  • You can do that either by screensharing and discussing or working individually.

  • When discussing other people’s projects please remember to be respectful and constructive. The goal of this exercise is not to criticize other projects but to learn from other projects and to collect the aspects that you enjoyed finding in a README and to also collect aspects which you have searched for but which are sometimes missing.

  • Please note observations and recommendations in the collaborative notes.

Sphinx and Markdown

In sphinx.md:

Sphinx-1: Generate the basic documentation template

Create a directory for the example documentation, step into it, and inside generate the basic documentation template:

$ mkdir doc-example
$ cd doc-example

We create the basic structure of the project manually.

File/directory

Contents

conf.py

Documentation configuration file

index.md

Main file in Sphinx

Let’s create the index.md with this content:

# Documentation example with Sphinx

A small example of how to use Sphinx and MyST 
to create easily readable and aesthetically pleasing
documentation.

```{toctree}
:maxdepth: 2
:caption: Contents:
some-feature.md
```

Note that indentation and spaces play a role here.

We also create a conf.py configuration file, with this content:

project = 'Test sphinx project'
author = 'Alice, Bob'
release = '0.1'            
                                                                                
extensions = ['myst_parser']
                                                                                
exclude_patterns = ['_build']

For more information about the configuration, see the Sphinx documentation.

Let’s create the file some-feature.md (in Markdown format) which we have just listed in index.md:

# Some feature

## Subsection

Exciting documentation in here.
Let's make a list (empty surrounding lines required):

- item 1

  - nested item 1
  - nested item 2

- item 2
- item 3

We now build the site:

$ ls -1

conf.py
index.md
some-feature.md

$ sphinx-build . _build

... lots of output ...
build succeeded.

The HTML pages are in _build.

$ ls -1 _build

_sources
_static
genindex.html
index.html
objects.inv
search.html
searchindex.js
some-feature.html

Now open the file _build/index.html in your browser.

  • Linux users, type:

    $ xdg-open _build/index.html

  • macOS users, type:

    $ open _build/index.html

  • Windows users, type:

    $ start _build/index.html

  • If the above does not work: Enter file:///home/user/doc-example/_build/index.html in your browser (adapting the path to your case).

Hopefully you can now see a website. If so, then you are able to build Sphinx pages locally. This is useful to check how things look before pushing changes to GitHub or elsewhere.

Note that you can change the styling by adding the line

html_theme = "<my favorite theme>"

in conf.py. For instance you can usesphinx_rtd_theme to have the Read the Docs look (make sure the sphinx_rtd_theme python package is available first)

In sphinx.md:

Sphinx-2: Add more content to your example documentation

  1. Add a entry below some-feature.md labeled another-feature.md (or a better name) to the index.md file.

  2. Create a file another-feature.md in the same directory as the index.md file.

  3. Add some content to another-feature.md, rebuild with sphinx-build . _build, and refresh the browser to look at the results.

  4. Use the MyST Typography page as help.

Experiment with the following Markdown syntax:

  • *Emphasized text* and **bold text**

  • Headings:

# Level 1

## Level 2

### Level 3

#### Level 4

  • An image: ![alt text](image.png)

  • [A link](https://www.example.org)

  • Numbered lists (numbers adjusted automatically):

1. item 1
2. item 2
3. item 3
1. item 4
1. item 5

  • Simple tables:

| No.  |  Prime |
| ---- | ------ |
| 1    |  No    |
| 2    |  Yes   |
| 3    |  Yes   |
| 4    |  No    |

  • Code blocks:

The following is a Python code block:
```python
  def hello():
      print("Hello world")
```

And this is a C code block:
```c
#include <stdio.h>
int main()
{
    printf("Hello, World!");
    return 0;
}
```

  • You could include an external file (here we assume a file called “example.py” exists; at the same time we highlight lines 2 and 3):

```{literalinclude} example.py
:language: python
:emphasize-lines: 2-3
```

  • Math equations with LaTeX should work out of the box. Try this (result below):

This creates an equation:
```{math}
a^2 + b^2 = c^2
```

This is an in-line equation, {math}`a^2 + b^2 = c^2`, embedded in text.

This creates an equation:

\[a^2 + b^2 = c^2\]

This is an in-line equation, \(a^2 + b^2 = c^2\), embedded in text.

Older versions of Sphinx

In some older versions, you might need to edit conf.py and add sphinx.ext.mathjax:

extensions = ['myst_parser', 'sphinx.ext.mathjax']

In sphinx.md:

Sphinx-3: Auto-generating documentation from Python docstrings

  1. Write some docstrings in functions and/or class definitions to a python module multiply.py:

def multiply(a: float, b: float) -> float:
    """
    Multiply two numbers.

    :param a: First number.
    :param b: Second number.
    :return: The product of a and b.
    """
    return a * b

  1. In the file conf.py add autodoc2 to the “extensions”, and add the list autodoc2_packages which will mention "multiply.py":

extensions = ['myst_parser', "autodoc2"]

autodoc2_packages = [
    "multiply.py"
]

If you already have extensions from another exercise, just add "autodoc2" to the existing list.

  1. Add apidocs/index to the toctree in index.md.

```{toctree}
:maxdepth: 2
:caption: Contents:

...
apidocs/index

  1. Re-build the documentation and check the “API reference” section.

In sphinx.md:

Sphinx-4: Writing Sphinx content with Jupyter

  1. For simplicity, create a text-based notebook files flower.md in the same directory as the index.md file. This file will be converted to a Jupyter notebook by the myst_nb Sphinx extension and then executed by Jupyter. Fill the file with the following content:

---
file_format: mystnb
kernelspec:
  name: python3
---
# Flower plot

```{code-cell} ipython3
import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots(1, 1, figsize=(5, 8), subplot_kw={"projection": "polar"})
theta = np.arange(0, 2 * np.pi, 0.01)
r = np.sin(5 * theta)
ax.set_rticks([])
ax.set_thetagrids([])
ax.plot(theta, r);
ax.plot(theta, np.full(len(theta), -1));
```

Note that there needs to be a title in the notebook (a heading starting with a single #), that will be used as an entry and link in the table of content. 2. In the file conf.py modify extensions to remove "myst_parser" and add "myst_nb" (you will get an error if you include both):

extensions = ["myst_nb"]

Note that MyST parser functionality is included in MyST NB, so everything else will continue to work as before.

  1. List flower in the toctree in index.md.

```{toctree}
:maxdepth: 2
:caption: Contents:
...
flower.md
```

  1. Re-build the documentation and check the “Flower” section.

  2. Alternatively, you can directly add .ipynb files saved from Jupyter notebook or Jupyter lab. Just make sure to list it in the toctree in index.md with the correct path.

If you have problems, consider cleaning manually the jupyter_execute directory.

Motivation and wishlist

In wishlist.md:

In wishlist.md:

Testing locally

In locally.md:

Local-1: Create a minimal example (15 min)

In this exercise, we will create a minimal example using the pytest, run the test, and show what happens when a test breaks.

  1. Create a new directory and change into it:

    $ mkdir local-testing-example
$ cd local-testing-example

  2. Create an example file and paste the following code into it

Create example.py with content

def add(a, b):
    return a + b

def test_add():
    assert add(2, 3) == 5
    assert add('space', 'ship') == 'spaceship'

This code contains one genuine function and a test function. pytest finds any functions beginning with test_ and treats them as tests.

  1. Run the test

$ pytest -v example.py

============================================================ test session starts =================================
platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/user/pytest-example, inifile:
collected 1 item

example.py::test_add PASSED

========================================================= 1 passed in 0.01 seconds ===============================

Yay! The test passed!

Hint for participants trying this inside Spyder or IPython: try !pytest -v example.py.

  1. Let us break the test!

Introduce a code change which breaks the code (e.g. - instead of +) and check whether our test detects the change:

$ pytest -v example.py

============================================================ test session starts =================================
platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/user/pytest-example, inifile:
collected 1 item

example.py::test_add FAILED

================================================================= FAILURES =======================================
_________________________________________________________________ test_add _______________________________________

    def test_add():
>       assert add(2, 3) == 5
E       assert -1 == 5
E         --1
E         +5

example.py:6: AssertionError
========================================================= 1 failed in 0.05 seconds ==============

Notice how pytest is smart and includes context: lines that failed, values of the relevant variables.

In locally.md:

(optional) Local-2: Create a test that considers numerical tolerance (10 min)

Let’s see an example where the test has to be more clever in order to avoid false negative.

In the above exercise we have compared integers. In this optional exercise we want to learn how to compare floating point numbers since they are more tricky (see also “What Every Programmer Should Know About Floating-Point Arithmetic”).

The following test will fail and this might be surprising. Try it out:

def add(a, b):
    return a + b

def test_add():
    assert add(0.1, 0.2) == 0.3

Your goal: find a more robust way to test this addition.

In locally.md:

Test design

In test-design.md:

Design-1: Design a test for a function that receives a number and returns a number

def factorial(n):
    """
    Computes the factorial of n.
    """
    if n < 0:
        raise ValueError('received negative input')
    result = 1
    for i in range(1, n + 1):
        result *= i
    return result

Discussion point: The factorial grows very rapidly. What happens if you pass a large number as argument to the function?

In test-design.md:

Design-2: Design a test for a function that receives two strings and returns a number

def count_word_occurrence_in_string(text, word):
    """
    Counts how often word appears in text.
    Example: if text is "one two one two three four"
             and word is "one", then this function returns 2
    """
    words = text.split()
    return words.count(word)

In test-design.md:

Design-3: Design a test for a function which reads a file and returns a number

def count_word_occurrence_in_file(file_name, word):
    """
    Counts how often word appears in file file_name.
    Example: if file contains "one two one two three four"
             and word is "one", then this function returns 2
    """
    count = 0
    with open(file_name, 'r') as f:
        for line in f:
            words = line.split()
            count += words.count(word)
    return count

In test-design.md:

Design-4: Design a test for a function with an external dependency

This one is not easy to test because the function has an external dependency.

def check_reactor_temperature(temperature_celsius):
    """
    Checks whether temperature is above max_temperature
    and returns a status.
    """
    from reactor import max_temperature
    if temperature_celsius > max_temperature:
        status = 1
    else:
        status = 0
    return status

In test-design.md:

Design-5: Design a test for a method of a mutable class

class Pet:
    def __init__(self, name):
        self.name = name
        self.hunger = 0
    def go_for_a_walk(self):  # <-- how would you test this function?
        self.hunger += 1

In test-design.md:

Design-6: Experience test-driven development

Write a test before writing the function! You can decide yourself what your unwritten function should do, but as a suggestion it can be based on FizzBuzz - i.e. a function that:

  • takes an integer argument

  • for arguments that are multiples of three, returns “Fizz”

  • for arguments that are multiples of five, returns “Buzz”

  • for arguments that are multiples of both three and five, returns “FizzBuzz”

  • fails in case of non-integer arguments or integer arguments 0 or negative

  • otherwise returns the integer itself

When writing the tests, consider the different ways that the function could and should fail.

After you have written the tests, implement the function and run the tests until they pass.

In test-design.md:

Design-7: Write two different types of tests for randomness

Consider the code below which simulates playing Yahtzee by using random numbers. How would you go about testing it?

Try to write two types of tests:

  • a unit test for the roll_dice function. Since it uses random numbers, you will need to set the random seed, pre-calculate what sequence of dice throws you get with that seed, and use that in your test.

  • a test of the yahtzee function which considers the statistical probability of obtaining a “Yahtzee” (5 dice with the same value after three throws), which is around 4.6%. This test will be an integration test since it tests multiple functions including the random number generator itself.

import random
from collections import Counter


def roll_dice(num_dice):
    return [random.choice([1, 2, 3, 4, 5, 6]) for _ in range(num_dice)]


def yahtzee():
    """
    Play yahtzee with 5 6-sided dice and 3 throws.
    Collect as many of the same dice side as possible.
    Returns the number of same sides.
    """

    # first throw
    result = roll_dice(5)
    most_common_side, how_often = Counter(result).most_common(1)[0]

    # we keep the most common side
    target_side = most_common_side
    num_same_sides = how_often
    if num_same_sides == 5:
        return 5

    # second and third throw
    for _ in [2, 3]:
        throw = roll_dice(5 - num_same_sides)
        num_same_sides += Counter(throw)[target_side]
        if num_same_sides == 5:
            return 5

    return num_same_sides


if __name__ == "__main__":
    num_games = 100

    winning_games = list(
        filter(
            lambda x: x == 5,
            [yahtzee() for _ in range(num_games)],
        )
    )

    print(f"out of the {num_games} games, {len(winning_games)} got a yahtzee!")

In test-design.md:

Design-8: Design (but not write) an end-to-end test for the uniq program

To have a tangible example, let us consider the uniq command. This command can read a file or an input stream and remove consecutive repetition. The program behind uniq has been written by somebody else, it probably contains some functions, but we will not look into it but regard it as “black box”.

If we have a file called repetitive-text.txt containing:

(all together now) all together now
(all together now) all together now
(all together now) all together now
(all together now) all together now
(all together now) all together now
another line
another line
another line
another line
intermission
more repetition
more repetition
more repetition
more repetition
more repetition
(all together now) all together now
(all together now) all together now

… then feeding this input file to uniq like this:

$ uniq < repetitive-text.txt

… will produce the following output with repetitions removed:

(all together now) all together now
another line
intermission
more repetition
(all together now) all together now

How would you write an end-to-end test for uniq?

In test-design.md:

Design-9: More end-to-end testing

  • Now imagine a code which reads numbers and produces some (floating point) numbers. How would you test that?

  • How would you test a code end-to-end which produces images?

In test-design.md:

Design-10: Create an actual end-to-end test

Often, you can include tests that run your whole workflow or program. For example, you might include sample data and check the output against what you expect. (including sample data is a great idea anyway, so this helps a lot!)

We’ll use the word-count example repository https://github.com/coderefinery/word-count.

As a reminder, you can run the script like this to get some output, which prints to standard output (the terminal):

$ python3 code/count.py data/abyss.txt

Your goal is to make a test that can run this and let you know if it’s successful or not. You could use Python, or you could use shell scripting. You can test if these two lines are in the output: the 4044 and and 2807.

Python hint: subprocess.check_output will run a command and return its output as a string.

Bash hint: COMMAND | grep "PATTERN" (“pipe to grep”) will be true if the pattern is in the command.

Automated testing

In continuous-integration.md:

Exercise CI-1: Create and use a continuous integration workflow on GitHub or GitLab

In this exercise, we will:

  • A. Create and add code to a repository on GitHub/GitLab (or, alternatively, fork and clone an existing example repository)

  • B. Set up tests with GitHub Actions/ GitLab CI

  • C. Find a bug in our repository and open an issue to report it

  • D. Fix the bug on a bugfix branch and open a pull request (GitHub)/ merge request (GitLab)

  • E. Merge the pull/merge request and see how the issue is automatically closed.

  • F. Create a test to increase the code coverage of our tests.

(Optional) Full-cycle collaborative workflow

In full-cycle-ci.md:

FullCI-1: Create and use a continuous integration workflow on GitHub or GitLab with pull requests and issues

This is an expanded version of the automated testing demonstration. The exercise is performed in a collaborative circle within the exercise group (breakout room).

The exercise takes 20-30 minutes.

In this exercise, everybody will:

A. Create a repository on GitHub/GitLab (everybody should use a different repository name for their repository) B. Commit code to the repository and set up tests with GitHub Actions/ GitLab CI C. Everybody will find a bug in their repository and open an issue in their repository D. Then each one will clone the repo of one of their exercise partners, fix the bug, and open a pull request (GitHub)/ merge request (GitLab) E. Everybody then merges their co-worker’s change

In full-cycle-ci.md:

(optional) FullCI-2: Add a license file to the previous exercise’s repository

In the Social coding and open software lesson we learn how important it is to add a LICENSE file.

Your goal:

  • You discover that your coworker’s repository does not have a LICENSE file.

  • Open an issue and suggest a LICENSE.

  • Then add a LICENSE via a pull/merge request, referencing the issue number.

Deploying Sphinx documentation to GitHub Pages

In gh_workflow.md:

GH-Pages-1: Deploy Sphinx documentation to GitHub or GitLab Pages

In this exercise we will create an example repository on a software forge (GitHub or GitLab) and deploy it to the corresponding “Page” service (GitHub pages or GitLab pages)

Preliminary: Verify the “Pages” feature is available on your forge

On github.com, the feature is typically available.

Step 1:

Generate the repository to be used as the starting point. Repositories can be generated from templates, created from imports, or created as empty and filled with a push from you own machine.

In the case of GitHub, the most convenient way is to generate the repository from a template.

Go to the documentation-example project template on GitHub and create a copy to your namespace.

  • Give it a name, for instance “documentation-example”.

  • You don’t need to “Include all branches”

  • Click on “Create a repository”.

Step 2: Browse the new repository.

  • The documentation part of the project is in doc/ (many projects do it this way).

  • The source code for your project could then go under src/.

Step 3: Automate the generation of the documentation

Add the GitHub Action to your new Git repository.

  • Add a new file at .github/workflows/documentation.yml (either through terminal or web interface), containing:

 1name: documentation
 2
 3on: [push, pull_request, workflow_dispatch]
 4
 5permissions:
 6  contents: write
 7
 8jobs:
 9  docs:
10    runs-on: ubuntu-latest
11    steps:
12      - uses: actions/checkout@v4
13      - uses: actions/setup-python@v5
14      - name: Install dependencies
15        run: |
16          pip install sphinx sphinx_rtd_theme myst_parser
17      - name: Sphinx build
18        run: |
19          sphinx-build doc _build
20      - name: Deploy to GitHub Pages
21        uses: peaceiris/actions-gh-pages@v3
22        if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
23        with:
24          publish_branch: gh-pages
25          github_token: ${{ secrets.GITHUB_TOKEN }}
26          publish_dir: _build/
27          force_orphan: true

  • You don’t need to understand all of the above – you should mainly pay attention the highlighted lines which are shell commands (we know this because they are part of a run: | section). The first uses pip to install the dependencies and the second runs sphinx-build to actually build the documentation (as we saw in the previous episode).

  • After the file has been committed (and pushed), check the action at https://github.com/USER/documentation-example/actions (replace USER with your GitHub username).

Step 4: Enable Pages feature and verify it’s running

  • Go to “Settings” -> “Pages”.

  • Under “Build and deployment”

    • In the Source section: choose “Deploy from a branch” in the dropdown menu

    • In the Branch section: choose “gh-pages” and “/ (root)” in the dropdown menus and click the Save button.

  • You should now be able to verify the pages deployment in the “Actions” list (this is how it looks like for this lesson material).

Step 5: Verify the result

Your site should now be live at https://USER.github.io/documentation-example/ (replace USER).

Step 6 (optional): Verify refreshing the documentation

  • Commit some changes to your documentation

  • Verify that the documentation website refreshes after your changes (can take few seconds or a minute)

In gh_workflow.md:

GH-Pages-2: Putting it all together

  1. Follow the above instructions to create a new repository with a Sphinx documentation project;

  2. Try adding one or more of the following to your Sphinx project:

    1. API documentation (see exercise in part 1 on API references) which requires the sphinx-autodoc2 package.

    2. a Jupyter notebook (see exercise in part 1 on Jupyter notebooks) which requires the myst-nb package.

    3. change the theme (see the end of the quickstart in part 1). You can browse themes and find their package names on the Sphinx themes gallery.

    Important

    The computer on which the GitHub actions run is not your local machine, and might not have the libraries you need to build the project. Make sure you update the dependencies (installed with pip in the demonstration) appropriately.

    Important

    Make sure the correct file paths are used. This will require adjusting paths from the example from the previous episode to the new layout. Note many paths, including e.g. the autodoc2_packages preference are now relative to the doc/ directory.

What do you need to change in the workflow file?

Hosting websites/homepages on GitHub Pages

In gh-pages.md:

GH-Pages-2: Host your own github page

  • Deploy own website reusing a template:

    • Follow the steps from GitHub Pages https://pages.github.com/. The documentation there is very good so there is no need for us to duplicate the screenshots.

    • Select “Project site”.

    • Select “Choose a theme”.

    • Follow the instructions on https://pages.github.com/.

    • Browse your page on https://USERNAME.github.io/REPOSITORY (adjust “USERNAME” and “REPOSITORY”).

  • Make a change to the repository after the webpage has been deployed for the first time.

  • Please wait few minutes and then verify that the change shows up on the website.