Use Procrastinate in a Django application

Many Django projects are deployed using PostgreSQL, so using procrastinate in conjunction with Django would remove the necessity of having another broker to schedule tasks, thereby reducing infrastructure costs.

It’s important to note that despite there’s support for Django inside procrastinate, there are still some pending issues to improve the Django experience - please feel free to contribute! Additionally, it’s worth noting that there are other Python job scheduling libraries based on postgres’ LISTEN/NOTIFY that integrate with Django. For instance, django-pgpubsub is more focused on Django, although it is still in the early stages of development.

To start, install procrastinate with:

$ (venv) pip install 'procrastinate[django]'

This tells pip to install procrastinate and consider the extra dependencies from the group of dependencies named django. For now, this group only contains Django itself, which you likely already have in your project’s dependencies. So why bother?

Specifying your dependency to the “django extras” will ensure that your Django version and the one we support stay in sync through time (for now, we support every version, but if we learn of strong incompatibilities, we’ll update the lib: we’re considering every version is compatible until proven otherwise). Also, while this is not the case today, if our Django integration ever requires other third-party packages, they will be added here.

Add procrastinate Django app to your INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    "procrastinate.contrib.django",
]

After that, you need to create a new directory to contain the procrastinate app, where you will define your tasks. This app will run independently of Django but will be able to access the whole Django ecosystem (it’s not a Django app, nor will be inside any of your existing Django apps). Create a new directory tasks and fill the tasks/__init__.py with:

import django
django.setup()  # Setup Django inside the worker so you can import/use ORM etc.

from asgiref.sync import sync_to_async  # Django's ORM is still sync
from procrastinate import App, AiopgConnector
from procrastinate.contrib.django import connector_params

from myapp.models import MyModel


app = App(connector=AiopgConnector(**connector_params()))
app.open()

@app.task(name="mytask")
async def mytask(obj_pk):
    @sync_to_async
    def work():
        print(f"Executing mytask for object {obj_pk}...")
        obj = MyModel.objects.get(pj=obj_pk)
        ...
    await work()

It is necessary to wrap all Django code with sync_to_async because Django ORM is still async. To avoid trouble with sync/async inside your tasks, we recommended you to create a “service” layer/module within your app and use the tasks only to call service code, so your tasks will be thin and maintenance will be easier, since the service layer could be tested as other modules of your app. To learn more about this approach, watch this talk at DjangoCon 2019.

To run the procrastinate worker properly, you need to set DJANGO_SETTINGS_MODULE to your project’s settings module and point to the tasks app you just created:

DJANGO_SETTINGS_MODULE=myproj.settings PYTHONPATH=. procrastinate --app=tasks.app worker

Note that tasks won’t be a Django app (so Django won’t import it), but you still need to be able to launch tasks from your Django code (for example, inside a view). Since the procrastinate app is not imported by Django, you must create a new app object accessible via Django to launch tasks (this object won’t act like a worker, it’s just your bridge from Django so you can launch tasks). Select an app and create myapp/tasks.py file with the following contents:

"""Expose procrastinate tasks so Django apps can call them"""

from procrastinate import App, Psycopg2Connector
from procrastinate.contrib.django import connector_params

# Depending on how the Django-postgres connection is configured, you may
# change the connector to `AiopgConnector`
app = App(connector=Psycopg2Connector(**connector_params()))
app.open()

# Task functions
mytask = app.configure_task(name="mytask")

(See Create your connector for more on how to instantiate your connector.)

Now you can finally launch this task from your myapp/views.py:

from myapp.tasks import mytask

def myview(request):
    ...
    mytask.defer(obj_pk=obj.pk)

Procrastinate comes with its own migrations so don’t forget to run ./manage.py migrate.