Python cron job with PM2 Docker

To run a python task periodically, we have many approaches. In this post, I use pm2 to manage the dockerized periodical python job.

Why PM2? PM2 provides an efficient way to manage processes or tasks.

Why Python, not NodeJS? In common, we use PM2 with NodeJS. However, PM2 can work with other kinds of interpreters. There is no magic behind the scene, I’d like to reuse my previous work.

By using PM2 as cron (aka schedule), we can remove unnecessary “while True” loop:

while True:
    doTask()
    sleepUntilNextTick()

The code is simple now:

doTask()

Firstly, write your python code as index.py

#index.py
import datetime
if __name__ == '__main__':
    print("hello", datetime.datetime.now())

Secondly, put you all required python package in requirements.txt. (I use requests here for example)

requests

Next, define the PM2 config file ecosystem.config.json. You can find all params here https://pm2.keymetrics.io/docs/usage/application-declaration/

{
    "apps": [{
        "name": "cron",
        "script": "index.py",
        "instances": "1",
        "exec_mode": "fork",
        "interpreter" : "python3",
        "cron_restart": "* * * * *",
        "watch": false,
        "autorestart": false
    }]
}

Take a look at two important keys:

• cron_restart: define the cron schedule. Check out this site if you are not familiar with the cron pattern https://crontab.guru/
• autorestart: must be false to prevent pm2 restarts the cron after completion immediately

Finally, define the Dockerfile

FROM keymetrics/pm2:16-alpine
# 1. add helpful packages
RUN apk add --no-cache --update alpine-sdk git wget python3 python3-dev ca-certificates musl-dev libc-dev gcc bash nano linux-headers && \
    python3 -m ensurepip && \
    pip3 install --upgrade pip setuptools
# 2. create user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
# 3. copy requirements.txt file to temp folder and installs required packages
RUN mkdir -p /home/appuser/temp
ADD ./requirements.txt /home/appuser/temp/requirements.txt
RUN pip3 install -r /home/appuser/temp/requirements.txt
# 4. copy project
RUN mkdir -p /home/appuser/project
COPY ./*.py /home/appuser/project/
COPY ./*.json /home/appuser/project/
# 5. grant permission
RUN chown appuser /home/appuser/project
# 6. setting enviroment
WORKDIR /home/appuser/project/
USER appuser
# 7. use pm2-logrotate
RUN pm2 install pm2-logrotate

1. Add helpful packages: some Alpine’s packages you should need, the most important packages are python3, python3-dev, libc-dev, gcc.
2. Create user: create a user in charge of manage docker container processes, avoid using the root account.
3. Install all required python packages from requirements.txt. Instead of copy all things once from your project to docker, the requirements.txt should be placed in a separate step to prevent docker from re-install python packages every you build the image.
4. Copy files of the project to the docker image.
5. Assign user permission to the project folder.
6. Set default path to project folder, and activate appuser as default container user.
7. Add-on: install pm2-logrorate to rotate the pm2 logs. More detail https://github.com/keymetrics/pm2-logrotate

Everything looks good now, let combine them into docker-compose.yml

version: "3.3"
services:
  app:
    build: ./project
    image: test-pm2-python-app:dev-1
    command: pm2-runtime start ecosystem.config.json
    restart: always
    logging:
      driver: "json-file"
      options:
        max-file: "10"
        max-size: "20m"

The command pm2-runtime start ecosystem.config.json will start your worker as scenario defined in ecosystem.config.json

Instead of using pm2 start, as PM2 document, we use pm2-runtime for the Docker environment. https://pm2.keymetrics.io/docs/usage/pm2-doc-single-page/

Wait, it doesn’t work yet

Very quickly, I found that the actual application behavior does not work as I expected. Once the cron script (index.py) is done, you will see the log

app_1  | 2021-09-19T17:26:16: PM2 log: App [cron:2] exited with code [0] via signal [SIGINT]
app_1  | 2021-09-19T17:26:22: PM2 log: 0 application online, retry = 3
app_1  | 2021-09-19T17:26:24: PM2 log: 0 application online, retry = 2
app_1  | 2021-09-19T17:26:26: PM2 log: 0 application online, retry = 1
app_1  | 2021-09-19T17:26:28: PM2 log: 0 application online, retry = 0
....
app_1  | 2021-09-19T17:26:28: PM2 log: PM2 successfully stopped
app_1 exited with code 2

It means the PM2 process will be terminated once there is no pm2-process alive, which makes the container stop then.

To solve this issue, we try to make an “infinite-loop dummy process” that keeps the PM2 process alive.

#dummy.py
import time
while True: time.sleep(1)

then, add the dummy process to ecosystem.config.json:

{
    "apps": [{
        "name": "dummy",
        "script": "dummy.py",
        "autorestart": true,
        "interpreter" : "python3",
        "exec_mode": "fork",
        "instances": "1"
    },
    {
        "name": "cron",
        "script": "index.py",
        "instances": "1",
        "exec_mode": "fork",
        "interpreter" : "python3",
        "cron_restart": "* * * * *",
        "watch": false,
        "autorestart": false
    }]
}

(Please lemme know if there is a better way to keep the PM2 process alive)

Besides cron mode, you can also use restart_delay to trigger the worker after a gap of time (similar to time.sleep)

Please check out the PM2 document for more information. Good luck!