Build automation, packaging, and release

Software Engineering

(for Intelligent Distributed Systems)

A.Y. 2024/2025

Giovanni Ciatto (reusing material made by Danilo Pianini)

Compiled on: 2025-06-30 — printable version

back

The build “life cycle”

(Not to be confused with the system development life cycle (SDLC))

The process of creating tested deployable software artifacts
from source code

May include, depending on the system specifics:

1. Source code manipulation and generation
2. Source code quality assurance
3. Dependency management
4. Compilation, linking
5. Binary manipulation
6. Test execution
7. Test quality assurance (e.g., coverage)
8. API documentation
9. Packaging
10. Delivery

Build automation

Automation of the build lifecycle

• In principle, the lifecycle could be executed manually
• In reality time is precious and repetitivy is boring (and error-prone)

$\Rightarrow$ Create software that automates the building of some software!

• All those concerns that hold for software creation hold for build systems creation…

Declarative build automation systems

Modern build automation systems are declarative

Declarative software $\approx$ the program describes what to do, not how to do it
(as opposed to imperative software $\approx$ the program describes how to do it,
e.g. Python)

In other words:

• they rely on conventions and defaults
• as long as developers follow them, the configuration is minimal
• good look if you try to deviate from conventions :)

Many modern languages (such as Rust) come with a build automator as part of their distribution.

In Python

Python is an interpreted language

• An arguably old one, too (1991)
• Initially used mainly for scripting
• No need to compile (as far as the project is pure Python)
• Build systems were a less pressing concerns than with other platforms…

The need for build systems in Python emerged with more complex use cases

• Since there were none, now there are several tools that do build-related jobs:

• The feature set varies wildly
• They are meant to solve different problems!

Python’s conflicting standards

Since there were no standard management systems originally, multiple tools proliferated

• The Python Packaging Authority (PyPA) is inconsistent in its suggestions:

  • Recommends venv
  • Also recommends Pipenv , which uses virtualenv
  • Also endorses Poetry

• Many Python developers also exploit PyEnv

• Many data scientists use Anaconda

From now on, let’s use Poetry

Poetry is a declarative tool for dependency management, packaging, and release in Python

• It handles both dependencies and dev-dependencies

  • replacing requirements.txt and requirements-dev.txt

• It automates the 1-project-1-Python-env convention

• It simplifies the packaging process for the project

• It simplifies the publication process on PyPI (or other software repositories)

Poetry’s canonical project structure

root-directory/
├── main_package/
│   ├── __init__.py
│   ├── sub_module.py
│   └── sub_package/ 
│       ├── __init__.py 
│       └── sub_sub_module.py 
├── test/
│   ├── test_something.py
│   └── test_something_else.py/ 
├── pyproject.toml              # File where project configuration (metadata, dependencies, etc.) is stored
├── poetry.toml                 # File where Poetry configuration is stored
├── poetry.lock                 # File where Poetry locks the dependencies
└── README.md

Notice that, w.r.t. the canonical project structure we have been using so far:

• the files requirements.txt and requirements-dev.txt are not present any more
• the files pyproject.toml, poetry.toml, and poetry.lock are new entries
• the poetry.lock file is generated automatically by Poetry, and you should not edit it

Example (pt. 1)

The calculator repository

1. Look at the pyproject.toml file

   [tool.poetry]
   # name of the package to be published
   name = "unibo-dtm-se-calculator"
   
   # files to be included for publication
   packages = [
       { include = "calculator" },
   ]
   
   # various metadata for publication
   version = "0.1.1"
   description = "A simple calculator toolkit written in Python, with several UIs. It is part of the Software Engineering course at the University of Bologna."
   authors = ["Giovanni Ciatto <giovanni.ciatto@unibo.it>"]
   license = "Apache 2.0"
   readme = "README.md"
   
   # dependencies (notice that Python is considered a dependency)
   [tool.poetry.dependencies]
   python = "^3.10.0"
   Kivy = "^2.3.0"
   
   # development dependencies
   [tool.poetry.group.dev.dependencies]
   poetry = "^1.7.0"
   pytest = "^8.1.0"
   coverage = "^7.4.0"
   mypy = "^1.9.0"
   
   # executable commands that will be created then installing this package
   [tool.poetry.scripts]
   calculator-gui = "calculator.ui.gui:start_app"
   calculator = "calculator.ui.cli:start_app"
   
   # where to download the dependencies from
   [[tool.poetry.source]]
   name = "PyPI"
   priority = "primary"
   
   # packaging dependencies
   [build-system]
   requires = ["poetry-core"]
   build-backend = "poetry.core.masonry.api"
   
   # mypy configuration
   [tool.mypy]
   ignore_missing_imports = true
   

   • learn how to specify dependencies here https://python-poetry.org/docs/dependency-specification/

Example (pt. 2)

The calculator repository

2. Look at the poetry.toml file

   # the project-specific environment will be created in the local .venv folder
   [virtualenvs]
   in-project = true 
   
   # packages produced by poetry may be published on the pypi-test repository
   [repositories.pypi-test]
   url = "https://test.pypi.org/legacy/"
   
   # another implicit repository is always available, namely PyPI
   # at https://pypi.org/
   

Exercise (pt. 1)

The calculator repository

1. Ensure you have Poetry installed on your system

   • https://python-poetry.org/docs/#installing-with-the-official-installer
   • run poetry --version to check

2. Fork the repository

3. Clone the repository on your system

   • git clone https://github.com/unibo-dtm-se/calculator.git
   • then, open the calculator folder into VS code

4. Run poetry install in the terminal. This shall:

   • create a virtual environment in the .venv directory
     • please ensure that a .venv directory exists in your project directory, after running this command
   • install the dependencies declared in the pyproject.toml file
   • lock the dependencies in the poetry.lock file

Dependency ranges and locking

• A project can depend on a specific version of a library or on a range of versions
• We want to be able to specify ranges, but retain the ability to use an exact version

This software is compatible with library version 2.3. For our examples, we used version 2.3.10

Expressing something like this is done via dependency locking:

• Configure the build file with the range of supported versions
• Use the build tool to lock the dependencies (pinpoint their version)
  • In practice, create a lock file where the exact version is explicit
• Locking usually also locks the transitive dependencies!
  • Once locked, we have a snapshot of a working environment

Poetry allows range specifications and locks automatically

Exercise (pt. 2)

The calculator repository

5. Let’s use a shell in the virtual environment created by Poetry

   • run poetry shell
   • if this operation was fine, you should be able to run the command calculator 1+1 to produce 2 as output
     • if you get an error, please let us know

6. Make sure that VS Code is using the same environment as the one created by Poetry.

   • Open the Command Palette (on / press Ctrl+⇧+P, use ⌘ instead of Ctrl on )
   • Type Python: Select Interpreter
   • Choose the local environment, i.e. the one having path ./.venv/bin/python

7. This is an ordinary project, where you can operate as usual

   • run python -m unittest discover -v -s tests in the terminal
   • try to run tests in the UI, too

Useful hints

1. So many Python environments in the shell… how to avoid mistakes?

   If you want to be 100% you’re running commands in the right environment, you can prefix them with poetry run:

   poetry run python -m unittest discover -v -s tests
   

2. Ok but what about VS Code’s environments?

   Get the habit of configuring the VS Code environment manually when working with Python projects

What’s the value added of Poetry?

It supports publishing.

Consider the following Web Page:
https://pypi.org/project/unibo-dtm-se-calculator

• This is the PyPI page of the unibo-dtm-se-calculator package

  • which we published by means of Poetry

• After the project is made available on PyPI, it can be installed by anyone

  • by running pip install unibo-dtm-se-calculator

Exercise

Installing a published package in a virgin environment

Let’s first create a new Python environment, and then install the unibo-dtm-se-calculator package in it

1. Create a new Python environment (to simulate the initially empty environment of a new developer/user)

   1. open a fresh new terminal
   2. create the fresh new environment python -m venv virgin-env
   3. activate the environment
   • on on / run source virgin-env/bin/activate
   • on use .\virgin-env\Scripts\Activate
   3. if you run commands calculator or calculator-gui they should fail

2. Install the unibo-dtm-se-calculator package pip install unibo-dtm-se-calculator

3. Try now to run the calculator in the shell

   • calculator 1+1 should produce 2 as output
   • calculao-gui should open a GUI calculator

How to publish Python packages on PyPI

Prerequisites

• Have an account on PyPI

  • generate an API token for your account

• Have an account on Test-PyPI

  • generate an API token for your account

Conceptual steps

1. Publish on Test-Pypi, to ensure the procedure works

2. Publish on PyPI

3. Profit

How to publish Python packages on Test-PyPI with Poetry

1. Build the project, i.e. produce installable package (only do this after all Q/A checks pass)

   • poetry build
   • this will create a .whl file and a .tar.gz file in the dist/ directory

2. Upload installable packages on Test-PyPI

   • poetry publish --repository pypi-test --username __token__ --password TEST_PYPI_API_TOKEN
     • the username is literally __token__
     • the password is the API token you generated on Test-PyPI
   • this will upload the package to Test-PyPI

3. That’s it!

   • visit the page of your package on Test-PyPI
   • try to install it in a virgin environment via pip install -i https://test.pypi.org/simple/ unibo-dtm-se-calculator

Problems that you may encounter

• Building may fail if your pyproject.toml is not properly configured, or missing data

• Upload may fail:

  • if your credentials are wrong, for some reason
  • if the package name is already taken
  • if the package version is already taken

• Installation may fail:

  • because your package has a dependency that is not available on Test-PyPI

How to publish Python packages on PyPI with Poetry

1. Build the project, i.e. produce installable package (only do this after all Q/A checks pass)

   • poetry build
   • this will create a .whl file and a .tar.gz file in the dist/ directory

2. Upload installable packages on PyPI

   • poetry publish --repository pypi-test --username __token__ --password PYPI_API_TOKEN
     • the username is literally __token__
     • the password is the API token you generated on PyPI
   • this will upload the package to PyPI

3. That’s it!

   • visit the page of your package on PyPI
   • try to install it in a virgin environment via pip install unibo-dtm-se-calculator

Problems that you may encounter

• Building may fail if your pyproject.toml is not properly configured, or missing data

  • but you should already have fixed this when publishing on Test-PyPI

• Upload may fail:

  • if your credentials are wrong, for some reason
  • if the package name is already taken
  • if the package version is already taken

• Installation may fail, but in this case there must be an issue in the package itself

Lessons learned

• Building a piece of software is not just writing code
• Software requires dependencies
• Dependency management is difficult
• All components of a runtime are part of the dependencies
  • Including interpreters
• Build Reproducibility is paramount
• Automatic configuration is desirable
• The Python build ecosystem is fragmented
  • Poetry is a modern take on Python dependency management and packaging
  • PyEnv can be used to let multiple Python versions live together

Check your understanding

• What is the build life cycle of a software project?
• In the context of Software Engineering, what is build automation?
• What can you expect, in general, from a build automation tool?
• In the context of build automation, what is dependency locking? Why is it necessary?
• In a Python project, what would you use Poetry for?
• Where would you release a Python project on Test PyPI? When on PyPI?
• What is Test PyPI? What’s its purpose?
• What are the steps to release a Python project via Poetry?

Lecture is Over

Compiled on: 2025-06-30 — printable version

back to ToC