Contributing

You’re welcome to come and procrastinate with us :)

TL;DR

$ source ./dev-env

Of course, feel free to read the script before launching it.

This script is intended to be a one-liner that sets up everything you need. It makes the following assumptions:

  • You’re using MacOS or Linux, and bash or zsh.

  • You already have python3 available

  • You either have virtualenv installed or your python3 supports -m venv (on Ubuntu, sudo apt install python3-venv)

  • Either you’ve already created a virtualenv, or you’re OK with the script creating a local virtualenv in .venv

  • Either you’ve already setup a PostgreSQL database and environment variables (PG*) are set or you have docker-compose available and port 5432 is free.

  • Either psql and other libpq executables are available in the PATH or they are located in usr/local/opt/libpq/bin (Homebrew).

The dev-env script will add the scripts folder to your path for the current shell, so in the following documentation, if you see scripts/foo, you’re welcome to call foo directly.

Instructions for contribution

Environment variables

The export command below will be necessary whenever you want to interact with the database (using the project locally, launching tests, …). These are standard libpq environment variables, and the values used below correspond to the Docker setup. Feel free to adjust them as necessary.

$ export PGDATABASE=procrastinate PGHOST=localhost PGUSER=postgres PGPASSWORD=password

Create your development database

The development database can be launched using Docker with a single command. The PostgreSQL database we used is a fresh standard out-of-the-box database on the latest stable version.

$ docker-compose up -d postgres

If you want to try out the project locally, it’s useful to have postgresql-client installed. It will give you both a PostgreSQL console (psql) and specialized commands like createdb we use below.

$ # Ubuntu
$ sudo apt install postgresql-client
$ createdb

$ # MacOS
$ brew install libpq
$ /usr/local/opt/libpq/bin/createdb

Set up your development environment

Install the package itself with development dependencies in a virtual environment:

$ python3 -m venv .venv
$ source .venv/bin/activate

You can check that your Python environment is properly activated:

(venv) $ which python
/path/to/current/folder/.venv/bin/python

Install local dependencies:

(venv) $ pip install -r requirements.txt

Run the project automated tests

With a running database:

(venv) $ pytest  # Test the code with the current interpreter

Or

$ tox  # Run all the checks for all the interpreters

If you’re not familiar with Pytest, do yourself a treat and look into this fabulous tool.

To look at coverage in the browser after launching the tests, use:

$ scripts/htmlcov

Keep your code clean

Before committing:

$ tox -e format,check-lint

If you’ve committed already, you can do a “Oops lint” commit, but the best is to run:

$ git rebase -i --exec 'tox -e format' origin/main

This will run all code formatters on each commits, so that they’re clean. If you’ve never done an interactive rebase before, it may seem complicated, so you don’t have to, but… Learn it, it’s really cool !

You can also use pre-commit which makes sure that all your commits are created clean:

scripts/bootstrap

This will keep you from creating a commit if there’s a linting problem.

In addition, an editorconfig file will help your favorite editor to respect procrastinate coding style. It is automatically used by most famous IDEs, such as Pycharm and VS Code.

Build the documentation

Build with:

$ scripts/docs  # build the html doc
$ scripts/htmldoc  # browse the doc in you browser

Run spell checking on the documentation (optional):

$ sudo apt install enchant
$ scripts/docs-spelling

Because of outdated software and version incompatibilities, spell checking is not checked in the CI, and we don’t require people to run it in their PR. Though, it’s always a nice thing to do. Feel free to include any spell fix in your PR, even if it’s not related to your PR (but please put it in a dedicated commit).

If you need to add words to the spell checking dictionary, it’s in docs/spelling_wordlist.txt. Make sure the file is alphabetically sorted.

If Sphinx’s console output is localized and you would rather have it in English, (which make google-based debugging much easier), use the environment variable export LC_ALL=C.utf-8

Migrations

Create database migration scripts

If you make changes to the database structure (procrastinate/sql/schema.sql) you also need to create a corresponding migration script in the procrastinate/sql/migrations directory.

For example, let’s say you want to add a column named extra to the procrastinate_jobs table. You will first edit procrastinate/sql/schema.sql and change the definition of the table to add that column. That would be sufficient for new Procrastinate users, but existing users, whose database already includes Procrastinate objects (tables, indexes, …), need to be able to migrate their existing schema into the new one. For that reason, as a Procrastinate developer, you’ll also need to create a migration script, whose content would look like this:

-- add a column extra to the procrastinate_jobs table
ALTER TABLE procrastinate_jobs ADD COLUMN extra TEXT;

The name of migration scripts must follow a specific pattern:

xx.yy.zz_ab_very_short_description_of_your_changes.sql

xx.yy.zz is the number of the latest released version of Procrastinate. (The latest release is the one marked Latest release on the Procrastinate releases page.) xx, yy and zz must be 2-digit numbers, with leading zeros if necessary. ab is the 2-digit migration script’s serial number, 01 being the first number in the series. And, finally, very_short_description_of_your_changes is a very short description of the changes (wow). It is important to use underscores between the different parts, and between words in the short description.

For example, let’s say the latest released version of Procrastinate is 1.0.1, and that the migrations directory already includes a migration script whose serial number is 01 for that release number. In that case, if you need to add a migration script, its name will start with 01.00.01_02_.

Backward-compatibility

As a Procrastinate developer, the changes that you make to the Procrastinate database schema must be compatible with the Python code of previous Procrastinate versions.

For example, let’s say that the current Procrastinate database schema includes an SQL function

procrastinate_func(arg1 integer, arg2 text, arg3 timestamp)

that you want to change to

procrastinate_func(arg1 integer, arg2 text)

The straightforward way to do that would be to edit the schema.sql file and just replace the old function by the new one, and add a migration script that removes the old function and adds the new one:

DROP FUNCTION procrastinate_func(integer, text, timestamp);
CREATE FUNCTION procrastinate_func(arg1 integer, arg2 text)
RETURNS INT
...

But if you do that you will break the Procrastinate Python code that uses the old version of the procrastinate_func function. The direct consequence of that is that Procrastinate users won’t be able to upgrade Procrastinate without incurring a service outage.

So when you make changes to the Procrastinate database schema you must ensure that the new schema still works with old versions of the Procrastinate Python code.

Going back to our procrastinate_func example. Instead of replacing the old function by the new one in schema.sql, you will leave the old function, and just add the new one. And your migration script will just involve adding the new version of the function:

CREATE FUNCTION procrastinate_func(arg1 integer, arg2 text)
RETURNS INT
...

The question that comes next is: when can the old version of procrastinate_func be removed? Or more generally, when can the SQL compatibility layer be removed?

The answer is some time after the next major version of Procrastinate!

For example, if the current Procrastinate version is 1.5.0, the SQL compatibility layer will be removed after 2.0.0 is released. The 2.0.0 release will be a pivot release, in the sense that Procrastinate users who want to upgrade from, say, 1.5.0 to 2.5.0, will need to upgrade from 1.5.0 to 2.0.0 first, and then from 2.0.0 to 2.5.0. And they will always migrate the database schema before updating the code.

The task of removing the SQL compatibility layer after the release of a major version (e.g. 2.0.0) is the responsibility of Procrastinate maintainers. More specifically, for the 2.1.0 release, Procrastinate maintainers will need to edit schema.sql and remove the SQL compatibility layer.

But, as a standard developer, when you make changes to the Procrastinate database schema that involves leaving or adding SQL statements for compatibility reasons, it’s a good idea to add a migration script for the removal of the SQL compatibility layer. This will greatly help the Procrastinate maintainers.

For example, let’s say the current released version of Procrastinate is 1.5.0, and you want to change the signature of procrastinate_func as described above. You will add a 1.5.0 migration script (e.g. 01.05.00_01_add_new_version_procrastinate_func.sql) that adds the new version of the function, as already described above. And you will also add a 2.0.0 migration script (e.g. 02.00.00_01_remove_old_version_procrastinate_func.sql) that takes care of removing the old version of the function:

DROP FUNCTION procrastinate_func(integer, text, timestamp);

In this way, you provide the new SQL code, the compatibility layer, and the migration for the removal of the compatibility layer.

Note

The migration scripts that remove the SQL compatibility code are to be added to the future_migrations directory instead of the migrations directory. And it will be the responsibility of Procrastinate maintainers to move them to the migrations directory after the next major release.

Migration tests

The continuous integration contains tests that will check that the schema and the migrations succeed in producing the same database structure. The migration tests are included in the normal test suite, but you can run them specifically with:

(venv) $ pytest tests/migration

Try our demo

With a running database, and its schema installed:

(venv) $ export PROCRASTINATE_APP=procrastinate_demo.app.app
(venv) $ procrastinate schema --apply

schedule some tasks with a script:

(venv) $ python -m procrastinate_demo

Or from the command line:

procrastinate defer procrastinate_demo.tasks.sum '{"a": 3, "b": 5}'
procrastinate defer procrastinate_demo.tasks.sum '{"a": 5, "b": 7}'
procrastinate defer procrastinate_demo.tasks.sum '{"a": 5, "b": "}")'
procrastinate defer procrastinate_demo.tasks.sum_plus_one '{"a": 4, "b": 7}'
procrastinate defer --lock a procrastinate_demo.tasks.sleep '{"i": 2}'
procrastinate defer --lock a procrastinate_demo.tasks.sleep '{"i": 3}'
procrastinate defer --lock a procrastinate_demo.tasks.sleep '{"i": 4}'
procrastinate defer procrastinate_demo.tasks.random_fail '{}'

Launch a worker with:

(venv) $ procrastinate worker

Use Docker for Procrastinate development

In the development setup described above, Procrastinate, its dependencies, and the development tools (tox, black, pytest, etc.) are installed in a virtual Python environment on the host system. Alternatively, they can be installed in a Docker image, and Procrastinate and all the development tools can be run in Docker containers. Docker is useful when you can’t, or don’t want to, install system requirements.

This section shows, through docker-compose command examples, how to test and run Procrastinate in Docker.

Build the procrastinate Docker image:

$ export UID GID
$ docker-compose build procrastinate

Run the automated tests:

$ docker-compose run --rm procrastinate pytest

Docker Compose is configured (in docker-compose.yml) to mount the local directory on the host system onto /src in the container. This means that local changes made to the Procrastinate code are visible in Procrastinate containers.

The UID and GID environment variables are set and exported for the Procrastinate container to be run with the current user id and group id. If not set or exported, the Procrastinate container will run as root, and files owned by root may be created in the developer’s working directory.

In the definition of the procrastinate service in docker-compose.yml the PROCRASTINATE_APP variable is set to procrastinate_demo.app.app (the Procrastinate demo application). So procrastinate commands run in Procrastinate containers are always run as if they were passed --app procrastinate_demo.app.app.

Run the procrastinate command :

$ docker-compose run --rm procrastinate procrastinate -h

Apply the Procrastinate database schema:

$ docker-compose run --rm procrastinate procrastinate schema --apply

Run the Procrastinate healthchecks:

$ docker-compose run --rm procrastinate procrastinate healthchecks

Start a Procrastinate worker (-d used to start the container in detached mode):

$ docker-compose up -d procrastinate

Run a command (bash here) in the Procrastinate worker container just started:

$ docker-compose exec procrastinate bash

Watch the Procrastinate worker logs:

$ docker-compose logs -ft procrastinate

Use the procrastinate defer command to create a job:

$ docker-compose run --rm procrastinate procrastinate defer procrastinate_demo.tasks.sum '{"a":3, "b": 5}'

Or run the demo main file:

$ docker-compose run --rm procrastinate python -m procrastinate_demo

Stop and remove all the containers (including the postgres container):

$ docker-compose down

Wait, there are async and await keywords everywhere!?

Yes, in order to provide both a synchronous and asynchronous API, Procrastinate needs to be asynchronous at core.

We’re using a trick to avoid implementing two almost identical APIs for synchronous and asynchronous usage. Find out more in the documentation, in the Discussions section. If you need information on how to work with asynchronous Python, check out:

Things that could be done more cleanly

As much as we’d like for our boilerplate to be perfect, there are still small things that can be improved:

  • Because ReadTheDocs doesn’t support installing package using poetry, the 2 possible ways of handling the problem would be to use extra dependencies for docs, or repeat our deps twice, we’ve gone ahead with the second solution. For now, it’s purely manual, though we should automate it.

  • Same with mypy dependencies in pre-commit, we need to duplicate them. Though this time, there is a script (scripts/typing-package-versions) that gives you the lines to paste in .pre-commit-config.yaml. It’s still expected to be done manually though.

Core contributor additional documentation

Issues

Please remember to tag Issues with appropriate labels.

Pull Requests

PR labels help release-drafter pre-fill the next release draft. They’re not mandatory, but releasing will be easier if they’re present.

Release a new version

There should be an active Release Draft with the changelog in GitHub releases. Make relevant edits to the changelog, (see TODO) including listing the migrations for the release. Click on Release, that’s it, the rest is automated.

When creating the release, GitHub will save the release info and create a tag with the provided version. The new tag will be seen by GitHub Actions, which will then create a wheel (using the tag as version number, thanks to our setup.py), and push it to PyPI (using the new API tokens). That tag should also trigger a ReadTheDocs build, which will read GitHub releases (thanks to our changelog extension) which will write the changelog in the published documentation (transformed from Markdown to RestructuredText).

After a new major version is released (e.g. 2.0.0), in preparation for the next minor release (2.1.0), the migration scripts in the future_migrations directory that remove the SQL compatibility code must be moved to the migrations directory. And the schema.sql file must be updated accordingly.

Note

If you need to edit the name or body of a release in the GitHub UI, don’t forget to also rebuild the stable and latest doc on readthedocs.