Technicalities of writing tests

(we will focus on Python, but the concepts are general)

1. the source code can now be conceinved as composed by two parts:

   • the main code: where the actual software is implemented
   • the test code: where the tests for the actual software are implemented

2. the test code is usually placed in a separate folder, and it is usually named tests/ (or test/)

3. the dependencies of the project are now of two sorts:

   • the main dependencies: the libraries required by the main code
   • the development (“dev”) dependencies: the libraries required by the quality assurance procedures
     • there exist several libraries which support testing, e.g. unittest (included in Python), or pytest (third-party)

4. developers may now want to launch not only the software, but also the tests

   • ad-hoc terminal commands or IDE plugins are available for this purpose

Updated project structure

root_directory/
├── main_package/               # main package (i.e. directory for the main code)
│   ├── __init__.py
│   ├── sub_module.py
│   └── sub_package/ 
│       ├── __init__.py 
│       └── sub_sub_module.py 
├── tests/                      # directory for the test code
│   ├── test_module_1.py
│   ├── ...
│   └── test_module_N.py 
├── .python-version
├── README.md
├── requirements-dev.txt        # file to list *development* dependencies
└── requirements.txt            # file to list *main* dependencies

Important conventions:

1. all the test code should be placed in a directory named tests/ (or test/)

2. the test code should be put into .py files whose name starts with test_

3. requirements.txt is for the main dependencies, requirements-dev.txt is for the dev dependencies

Kivy>=2.3.0

-r requirements.txt
pytest>=8.1.0

Nomenclature about testing

• System under test (SUT): the component of the software that is being tested

  • e.g. a class, a function, a module

• Test case: a class that contains the test functions for a specific SUT

  • each test case corresponds to one or more testing procedures for the same SUT
  • in case of multiple procedures, all must share the same set up or tear down activities
    • i.e. activities to before or after each testing procedure from the same test case

• Test suite: a collection of test cases, commonly related to similar SUTs

  • it commonly consists of a module, e.g. a test_*.py file

• Assertion: a boolean (i.e. either True or False) check about the SUT

  • if the assertion is True, the assertion passes, and the test proceeds
  • if the assertion is False, the test fails, and it is interrupted

• Test procedure: a sequence of actions and assertions about some SUT

  • it succeeds if all the assertions are True and no unexpected error occurs
  • it fails otherwise

Writing tests in Python

We adopt unittest, a built-in library for writing tests in Python

• it is inspired by the JUnit library for Java
• it is not the only one: pytest is a popular alternative (but it needs to be installed)

Anatomy of a test suite in unittest

Let’s assume this is the test_my_system.py test suite (full code here)

⬇️

Technicalities of unittest tests suites

• Many assertion functions, cf.: https://docs.python.org/3/library/unittest.html#assert-methods

• Many options to customise/parametrise your test suites, cf. https://docs.python.org/3/library/unittest.html

• How to run tests:

  • from the terminal: python -m unittest discover -v -s tests
    • where -v stands for verbose (i.e. more detailed output)
    • where -s stands for start directory (i.e. the directory where the tests are, in this case tests)
  • from an IDE: usually there is a dedicated button
  • from VS Code: there is a dedicated section which requires configuration

• Effect of running all tests with subcommand discover:

  • all the test_*.py files in the tests/ directory (and its sub-directories) are loaded
    • all sub-classes of unittest.TestCase from those files are instantiated
      • all the functions from those classes that start with test_ are executed
        1. the setUp function is executed before each test function
        2. the tearDown function is executed after each test function

Hands-on (pt. 1)

Playing a bit with unittest

Restoring dev dependencies

1. Fork the following repository: https://github.com/unibo-dtm-se/testable-calculator

2. Clone the forked repository on your machine

   • `git clone https://github.com/YOUR_GITHUB_USERNAME/testable-calculator

3. Open VS Code into the testable-calculator directory

   • let’s use VS Code’s integrated terminal from now on

4. Restore both dependencies and dev-dependencies

   • pip install -r requirements-dev.txt

Hands-on (pt. 2)

Playing a bit with unittest

Running tests via the terminal

5. Run the tests via the terminal

   • Minimalistic: python -m unittest discover -s tests

     ..............
     ---------------------------------------------------------------------- 
     Ran 14 tests in 0.478s
     
     OK
     

     (each dot represents a successful test procedure… not really clear, right?)

   • Verbose: python -m unittest discover -v -s tests (notice option -v)

     test_cli_with_invalid_expression (test_cli.TestCalculatorCli.test_cli_with_invalid_expression) ... ok
     test_cli_with_single_expression (test_cli.TestCalculatorCli.test_cli_with_single_expression) ... ok
     test_cli_with_sliced_expression (test_cli.TestCalculatorCli.test_cli_with_sliced_expression) ... ok
     [...]
     test_expression_insertion (test_model.TestCalculatorUsage.test_expression_insertion) ... ok
     
     ----------------------------------------------------------------------
     Ran 14 tests in 0.447s
     
     OK
     

     (one test per line: clearer)

Hands-on (pt. 3)

Playing a bit with unittest

Running tests via VS Code

(if you cannot find the Test section, look at the next slide)

What if you cannot find the Test section? (pt. 1)

You probably have and old version of VS Code, and you should update it

⬇️ Meanwhile, you can follow this workaround ⬇️

What if you cannot find the Test section? (pt. 2)

1. Go to the Extensions section of VS Code

   • you can do this by clicking on the Extensions icon in the Activity Bar on the side of the window

2. In the search bar of the Extensions section, type python tests

   • the first result should be the Python extension by Microsoft

   • the second result should be the Python Test Explorer extension by Little Fox Team

     

What if you cannot find the Test section? (pt. 3)

3. Click on the Install button of both extensions

   • while installing, VS Code may look like this

     

What if you cannot find the Test section? (pt. 4)

4. Once the installation is complete, you should see the Test section in the Activity Bar on the side of the window

   

Hands-on (pt. 4)

Playing a bit with unittest

Inspecting a real unit test

6. Have a look to the tests/test_model.py file and listen to the teacher explanation
   • it contains a test suite for the Calculator class

⬇️

Hands-on (pt. 4)

Playing a bit with unittest

Failing tests

7. Try to run tests via the terminal and via VS Code

   • notice that in VS Code you can run tests selectively

8. Let’s now simulate the scenario where tests are failing (e.g. due to buggy code)

   • edit the Calculator in file calculator/__init__.py to introduce a bug
     • e.g. change the __init__ function as follows:

       def __init__(self):
           self.expression = "0" # bug: the expression is not initially empty
       

9. Run the tests again: many tests should now fail

   • notice how the tests failure is reported in the terminal and in VS Code
   • try to spot the source of the problem, from the error reports

Hands-on (pt. 5)

Playing a bit with unittest

Testing the GUI (+ integration with model)

6. Have a look to the tests/test_gui.py file and listen to the teacher explanation:

• it contains a test suite for the CalculatorApp class

7. Notice that tests are based on custom _base class _(namely CalculatorGUITestCase), which adds
   • custom action (e.g. press_button(button_name))
   • custom assertions (e.g. assert_display(expected_text))
   • custom setup and teardown activities (showing / closing the GUI)

⬇️

Hands-on (pt. 5)

Playing a bit with unittest

Testing the GUI (+ integration with model)

6. Have a look to the tests/test_gui.py file and listen to the teacher explanation:

• it contains a test suite for the CalculatorApp class

7. Notice that tests are based on custom _base class _(namely CalculatorGUITestCase), which adds

   • custom action (e.g. press_button(button_name))
   • custom assertions (e.g. assert_display(expected_text))
   • custom setup and teardown activities (showing / closing the GUI)

8. In particular, have a look to the TestExpressions test case, and listen to the teacher explanation

⬇️

Interesting things to notice (pt. 1)

• To enable testing the GUI, the CalculatorApp class’s public API has been extended with further functionalities:
  • find_button_by(text): a function returning the button widget with the given text
  • display: an attribute referencing the display widget (it’s now public)

New entries

• find_button_by(text: str): is necessary to make simulate buttons pressure in the tests
• _browse_children(container): is a private functionality, necessary to implement find_button_by
• display: is necessary to make assertions about the displayed text in the tests

Interesting things to notice (pt. 2)

How these novel functionalities are implemented in practice is not that relevant, but here it is:

class CalculatorApp(App):
    # returns a sort of list of all the widgets directly or indirectly contained in the given container
    def _browse_children(self, container):
        yield container
        if hasattr(container, 'children'):
            for child in container.children:
                yield from self._browse_children(child)
    
    # returns the first widget in the GUI which 1. is a button and 2. whose text is equal to the given one
    def find_button_by(self, text) -> Button:
        for widget in self._browse_children(self.root):
            if isinstance(widget, Button) and widget.text == text:
                return widget
            
    def build(self):
        # ... (unchanged)
        self.display = Label(text="0", font_size=24, size_hint=(3, 1))
        # ... (unchanged)

    # the rest of the class is unchanged

Interesting things to notice (pt. 3)

Take-away: when writing post-hoc tests (i.e., after the main code has been already written), it is often necessary to extend the public API of the SUT to make its internal state and functioning observable and controllable from the outside, and therefore testable

• to avoid this situation, it’s paramount to write tests before the main code is written
  • we’ll see how

Interesting things to notice (pt. 4)

If you read them at the adequate abstraction level, each test case is telling a story about the SUT

• e.g. TestCalculatorMethods is telling the story of the Calculator class

  • it’s telling the story of how a Calculator object looks like when it is freshly instantiated
  • it’s telling the story of how a Calculator object behaves when it is used to build an expression
  • it’s telling the story of how a Calculator object behaves when it is used to evaluate an expression

• e.g. TestExpressions is telling the story of the CalculatorApp class (i.e. the GUI)

  • it’s telling the story of how the GUI looks like when it is freshly instantiated
  • it’s telling the story of how the GUI behaves when it is used to build an expression
  • it’s telling the story of how the GUI behaves when it is used to evaluate an expression

Take-away: the story you can picture in your mind when reading a test is a way to describe the test plan, the designer of the test suite was envisioning when writing the tests

Exercise

Write your own test suite

1. Focus on the test_gui.py file

2. Add one more test case for the GUI, say TestLayout, which ensures that:

   • the GUI has a display, and its initial text is 0
   • the GUI contains all the buttons for the digits, the operations, and the special commands
     • namely: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, +, -, *, /, =, C, .

3. Hints:

   • you may want to reuse the CalculatorGUITestCase class and its functionalities
   • you may want to add one more custom assertion, say assert_button_exists, to CalculatorGUITestCase
     • in order to check that a button with a given text is present in the GUI

One possible solution is in the next slide

⬇️ (please resist the temptation, and try to solve the exercise before looking at the solution) ⬇️

One possible solution

(also available on branch exercises/01-test-layout of the testable-calculator repository)

class CalculatorGUITestCase(unittest.TestCase):
    # rest of the class is unchanged 

    def assert_button_exists(self, button_text):
        self.assertIsNotNone(self.app.find_button_by(button_text))


class TestLayout(CalculatorGUITestCase):
    buttons_to_test = {
        'C',
        '7', '8', '9', '/',
        '4', '5', '6', '*',
        '1', '2', '3', '-',
        '.', '0', '=', '+',
    }

    def test_initial_display(self):
        self.assert_display("0")

    def test_buttons(self):
        for button_text in self.buttons_to_test:
            with self.subTest(button=button_text):
                self.assert_button_exists(button_text)

• what’s the purpose of subTest?
  • it easies the debugging of the test suite, in case of multiple failures
    • try launching tests from VS Code’s UI

Exercise

Adding features to the Calculator via TDD

Customers ask for new features in the calculator:

1. possibility to write expressions with parentheses (e.g. (1+2)*3)
2. possibility to write expressions with the square root function (e.g. sqrt(4))
3. possibility to write expressions with the power function (e.g. 2**3)

1. Extend the model’s test suite (i.e. file test_model.py which aims at testing the Calculator class)

   • this implies extensions for the Calculator class’s public API should be envisioned (not realised)
   • use your imagination to invent reasonable extensions

2. Extend the GUI’s test suite (i.e. file test_gui.py which aims at testing the CalculatorApp class, i.e. the GUI)

   • this implies novel buttons and actions should be envisioned (not realised) for the GUI
   • use your imagination to invent reasonable buttons and their effects

3. Launch your tests: it’s OK if novel tests fail at this stage

   • let’s just ensure they fail for the correct reasons (missing methods, or missing buttons)

One possible solution

(also available on branch exercises/02-tdd-before-impl of the testable-calculatorrepository)

Test suite for the model (i.e. test_model.py)

# other test cases are unchanged

class TestComplexExpressions(unittest.TestCase):
    def setUp(self):
        self.calculator = Calculator()

    def test_expression_with_parentheses(self):
        self.calculator.open_parenthesis()
        self.calculator.digit(1)
        self.calculator.plus()
        self.calculator.digit(2)
        self.calculator.close_parenthesis()
        self.calculator.multiply()
        self.calculator.digit(3)
        self.assertEqual("(1+2)*3", self.calculator.expression)
        self.assertEqual(9, self.calculator.compute_result())

    def test_expression_with_sqrt(self):
        self.calculator.digit(1)
        self.calculator.plus()
        self.calculator.square_root()
        self.calculator.open_parenthesis()
        self.calculator.digit(1)
        self.calculator.digit(1)
        self.calculator.minus()
        self.calculator.digit(2)
        self.calculator.close_parenthesis()
        self.assertEqual("1+sqrt(11-2)", self.calculator.expression)
        self.assertEqual(4, self.calculator.compute_result())

    def test_expression_with_pow(self):
        self.calculator.open_parenthesis()
        self.calculator.digit(1)
        self.calculator.plus()
        self.calculator.digit(1)
        self.calculator.close_parenthesis()
        self.calculator.power()
        self.calculator.digit(3)
        self.assertEqual("(1+1)**3", self.calculator.expression)
        self.assertEqual(8, self.calculator.compute_result())

One possible solution

(also available on branch exercises/02-tdd-before-impl of the testable-calculatorrepository)

Test suite for the GUI (i.e. test_gui.py)

class TestExpressions(CalculatorGUITestCase):
    # other test methods are unchanged

    def test_expression_with_parentheses(self):
        self.press_button("(")
        self.press_button("1")
        self.press_button("+")
        self.press_button("2")
        self.press_button(")")
        self.press_button("*")
        self.press_button("3")
        self.assert_display("(1+2)*3")
        self.press_button("=")
        self.assert_display("9")

    def test_expression_wit_sqrt(self):
        self.press_button("sqrt")
        self.press_button("4")
        self.press_button(")")
        self.assert_display("sqrt(4)")
        self.press_button("=")
        self.assert_display("2.0")

    def test_expression_with_pow(self):
        self.press_button("2")
        self.press_button("**")
        self.press_button("3")
        self.assert_display("2**3")
        self.press_button("=")
        self.assert_display("8")

Exercise (pt. 1)

Compute test coverage in Python

0. Notice the coverage (dev) dependency in the requirements-dev.txt file

   • it is a tool that measures line-of-code coverage in Python, documentation here
   • you should have restored dependencies already, if not pip install -r requirements-dev.txt

1. Runs the tests while measuring coverage: coverage run -m unittest discover -v -s tests/

   • same command as for running tests, but with coverage run instead of python
   • if not working, try python -m coverage run -m unittest discover -v -s tests/

2. Check the coverage report in the terminal coverage report -m

   • the output should be similar to the one below:

     Name                     Stmts   Miss  Cover   Missing
     ------------------------------------------------------
     calculator\__init__.py      38      3    92%   13, 39, 48
     calculator\ui\cli.py        21      3    86%   16-17, 27
     calculator\ui\gui.py        57      8    86%   55-57, 61, 63, 65, 69, 76     
     tests\test_cli.py           15      0   100%
     tests\test_gui.py           30      0   100%
     tests\test_model.py         38      0   100%
     ------------------------------------------------------
     TOTAL                      199     14    93%
     

     pretty obscure, isn’t it?

Exercise (pt. 2)

Compute test coverage in Python

3. Let’s try to create a more pleasant report, in HTML format: coverage html

   • this command will create a htmlcov folder in the current directory
   • the htmlcov/index.html file is a static Web page, reporting the coverage of your project

4. Open the htmlcov/index.html file in your browser (any of the following may work)

   • start .\htmlcov\index.html
   • open htmlcov/index.html
   • xdg-open htmlcov/index.html
   • double click on the htmlcov/index.html file in your Explorer / Finder / Dolphin or Nautilus etc

Exercise (pt. 4)

Compute test coverage in Python

5. You should see an overview similar to the terminal one

   

Exercise (pt. 5)

Compute test coverage in Python

6. If you click on a file, you may get a line-by-line report of test coverage

   

Quality Assurance: syntactical correctness

Syntactical correctness is the first level of quality assurance:

“Is the code well-formed, i.e. readable for a computer?

• In compiled languages, the compiler checks for syntactical correctness

  • raising an error if the code is not syntactically correct
  • most commonly linking errors are detected too by the compiler
    • e.g. calling a function which is not correctly imported

• Python is an interpreted language, so there is no compiler

  • but the CPython interpreter (the most frequently used) supports translation into bytecode
  • similarly to compiled languages, translation into bytecode implies syntax and linking checks

• Syntactical correctness can be checked in Python by means of:

  • the compileall standard ($\approx$ included in Python by default) module
    • triggered by python -m compileall CODE_DIRECTORY_1 CODE_DIRECTORY_2 ...

Example

Checking for syntactical correctness in Python

1. In the testable-calculator project, run python -m compileall calculator tests

   • you may notice an output similar to the following one:

     Listing 'calculator'...
     Listing 'calculator\\ui'...
     Listing 'tests'...
     

     which means that all .py files in those directories were checked, and they are syntactically correct

     • if there were any errors, they would be listed here

2. Try to artificially add some syntax error in some Python file

   • e.g. remove a closing parenthesis, or a colon

3. Run python -m compileall calculator tests again

   • this time, the log should make you aware of where the error is

Example (pt. 1)

Static analysis in Python with mypy

1. Notice the mypy (dev) dependency in the requirements-dev.txt file

   • it is a static analysis tool for Python, documentation here
   • you should have restored dependencies already, if not pip install -r requirements-dev.txt

2. Run mypy on the testable-calculator project

   • mypy calculator tests
   • you may notice an output similar to the following one:

     calculator\__init__.py:13: error: Unsupported operand types for + ("str" and "int")  [operator]
     calculator\__init__.py:44: error: Parameterized generics cannot be used with class or instance checks  [misc]
     calculator\__init__.py:44: error: Argument 2 to "isinstance" has incompatible type "<typing special form>"; expected "_ClassInfo"  [arg-type]
     calculator\__init__.py:46: error: Returning Any from function declared to return "int | float"  [no-any-return]
     Found 4 errors in 1 file (checked 6 source files)
     

Example (pt. 2)

Static analysis in Python with mypy

3. What are those errors?

   • the calculator works, and it passes tests, so they are not errors in the usual sense
   • they are flawed programming patterns, i.e. potential bugs
     • or at least potential sources of headaches

4. Listen to the teacher explanation about the meaning of those errors

   • and how to fix them