Task management

Tasks are handled by django-q.

For that, we need a django-q “cluster”. Right now we use one node. These are the related files

File	Purpose	Environment
start-django.sh	Starts RSR in the container	Local
start-django.sh	Starts RSR in the container	Prod / Test
docker-compose.override.yaml	Local worker service	Local
deployment.yaml	K8s container	Prod / Test
41-django-q.conf	Configuration file	All
worker.sh	Health probe	All

Starting up

The worker container is started with the same script as the web container. The only difference is the IS_WORKER env variable.

That starts

• probe server

• django-q cluster

Probe server

This is a simple HTTP server to quickly query the status of the cluster. It was written for two reasons:

• the django-q cluster provides no script to query its status

• django-admin commands are excruciatingly slow (5-10) seconds startup time). Too slow for a probe

It’s started with akvo.rsr.management.commands.django_q_probettp.

The server is probed by docker and k8s using the worker.sh probe

Unique tasks

Recurring tasks (that were earlier cronjobs) that shouldn’t be run in parallel aren’t natively supported by django-q. A custom solution was built within RSR that uses a cache heartbeat.

CacheHeartbeat is the relevant class which is used by a decorator unique_task.

In principle all it does is check if the heartbeat key exists in the cache to exit the wrapped function.